When the file is not required to be stored permanently but big, S3 might not be the best choice. I recently had to work on a video-related project that requires downloading and analyzing but doesn’t need to be stored after analysis. In our case, the key requirement was accessing files without using S3 over the different EC2 instances. Therefore, we decided to use EFS as a Network File System (NFS). OS: ubuntu 18.04 1. Launch EFS From AWS Management Console,…
Environment: Ubuntu 18.04 Error 1. OSError: mysql_config not found Solution 1 Error 2. unable to execute ‘x86_64-linux-gnu-gcc’: No such file or directory Solution 2 Error 3. MySQLdb/_mysql.c:46:10: fatal error: Python.h: No such file or directory Solution 3 Final Code Happy Coding!
In this post, we will talk about how to connect the AWS API gateway to EC2. Normally, we use the API gateway to make an API when we are using Lambda. However, for some cases, you might want to use an API gateway as a proxy for the EC2 instance server. Let’s say we have an EC2 instance running the Flask application that we installed from the last post. We want to make an API that is pointing the server…
![[:en]Docker[:]](https://i2.wp.com/codethief.io/wp-content/uploads/2018/09/27a11-jon-eckert-562977-unsplash.jpg?resize=1038%2C576&ssl=1)
When you deploy your React.js application through Docker, you can deploy with Nginx. The basic principle is, first build a React.js application and move the generated files to “share/nginx/html” in the container. You are just replacing the default Nginx welcoming page to your React.js application. Now you just need to run Nginx. Please check the Docker file below. Check the sample project
In this post, we will build the multi-node Hadoop cluster using three EC2 instances ( one for master, two for slaves). (I will assume that you know how to use AWS. If you don’t, please check this link) To run Map-Reduce task properly, you need enough memory. Therefore, we will use t2.medium type instance. (If you are a student and need some free credit, check this link.) AWS EC2 t2.medium×3 (1 for a name node, 2 for data nodes) Ubuntu…
When you need to launch multiple containers that work as a single application, it is difficult to manage with a single Dockerfile. That’s why we need to use “docker compose” so that we can manage our containers from a higher level. The application that we will test is a simple Django application. The database is PostgreSQL, and it will also be installed using Docker. Normally, it is not recommended to install the database using Docker. The database should be running…
Hello! We have set up the Hadoop environment from the previous post. And YES! It IS a hassle unless you need your own tuned version of the environment. Therefore, I’ll introduce a more convenient way to use Hadoop environment from this post. We’ll test MRjob or PySpark using AWS EMR. In part 1 we’ll launch the EMR and use it very naively (static instances and using HDFS). From part 2 we’ll use EMR more correctly (?) (using AWS CLI and…
Ubuntu version : 18.04Python version: 3.6.7 Let’s get started from Ubuntu. If you are in ubuntu server and type below command you will see the version of your current python installation. In my case, it was Python 3.6.9. Now we need to install ‘virtualenv’ module.(There are several approaches that we can set a virtual environment for python, but it seems like this one is recommended these days.) Then, we make our project directory. In side the project directory, we need…