In this post, we will build the multi-node Hadoop cluster using three EC2 instances ( one for master, two for slaves). (I will assume that you know how to use AWS. If you don’t, please check this link) To run Map-Reduce task properly, you need enough memory. Therefore, we will use t2.medium type instance. (If you are a student and need some free credit, check this link.) AWS EC2 t2.medium×3 (1 for a name node, 2 for data nodes) Ubuntu…
When you need to launch multiple containers that work as a single application, it is difficult to manage with a single Dockerfile. That’s why we need to use “docker compose” so that we can manage our containers from a higher level. The application that we will test is a simple Django application. The database is PostgreSQL, and it will also be installed using Docker. Normally, it is not recommended to install the database using Docker. The database should be running…
Hello! We have set up the Hadoop environment from the previous post. And YES! It IS a hassle unless you need your own tuned version of the environment. Therefore, I’ll introduce a more convenient way to use Hadoop environment from this post. We’ll test MRjob or PySpark using AWS EMR. In part 1 we’ll launch the EMR and use it very naively (static instances and using HDFS). From part 2 we’ll use EMR more correctly (?) (using AWS CLI and…
Ubuntu version : 18.04Python version: 3.6.7 Let’s get started from Ubuntu. If you are in ubuntu server and type below command you will see the version of your current python installation. In my case, it was Python 3.6.9. Now we need to install ‘virtualenv’ module.(There are several approaches that we can set a virtual environment for python, but it seems like this one is recommended these days.) Then, we make our project directory. In side the project directory, we need…
The answer is: ‘/’ Sometimes when you change your client-side code, and suddenly you experience a 301 redirect response from your server-side. In my case, I was using Django as a serverside and use ReactJS as a client-side. The API was implemented in a RESTful style. Most of the cases, the reason was I forgot to end with ‘/’ in my request URL. (Good) https://api.example.com/api/100/ (Bad) https://api.example.com/api/100 I haven’t really investigated what is behind this rule but, don’t panic and…
Python is a perfect programming language unless you have to deal with asynchronous action. Many programmers using Javascript complain about ‘callback hell’. Still, people become grateful when they have to deal with asynchronous action (simultaneous, parallel, concurrent, etc.). Recently I needed to send several requests at the same time, instead of waiting for the previous request to finish. Moreover, all responses should be collected as one variable. 1. Ordinary request If we call 10 requests sequentially, it took 11.285 seconds….
![[:ko]Photo by pan zhen on Unsplash[:]](https://i0.wp.com/codethief.io/wp-content/uploads/2020/04/pan-zhen-Hvjx3cywvM4-unsplash.jpg?resize=1038%2C576&ssl=1)
Youtube-dl can download videos in most situations; however, you may want to set headers. In order to do that, you need to import youtube_dl.utils. Here is the example!