![]() Even so, the alternatives - such as Vagrant - are generally worse. ![]() I personally find it tough to debug and its aggressive caching makes both cache busting and resource clearing difficult. Surprisingly it works on Windows - casually, even.īrief sidebar: Docker isn't a silver bullet, and honestly it's kind of a pain in the butt. The alternative, and the one I'm going to demo in this post, is to use Docker.ĭocker is a tool for managing Linux containers, which are a little like virtual machines without the virtualization, making them act like self-contained machines but much more lightweight than a full VM. Given that this is a fully operational Ubuntu environment, any tutorial that you follow for Ubuntu should also work in this environment. I have WSL 2 installed, which is faster and better in many ways aside but which (until recently? unclear) needs an insider build of Windows. Type "Ubuntu" into the search bar, mash enter, and be dumped into a containerized Linux environment.Install Ubuntu from the Microsoft Store.Install the WSL by running some cryptic PowerShell commands.Basically, steps look something like this: WSL stands for the " Windows Subsystem for Linux", and it's actually really cool. Luckily, there are two fairly sensible alternate approaches to consider which will let you run Airflow on a Windows machine: WSL and Docker. So running Airflow in Windows natively is dead in the water, unless you want to spend a bunch of months rewriting a bunch of the logic and arguing with the maintainers**. I was flabbergasted to find that casually trying to run Airflow on Windows resulted in a bad shim script, a really chintzy pathing bug, a symlinking issue* and an attempt to use the Unix-only passwords database. Unfortunately, Airbnb handed us a pathologically non-portable codebase. For data, Anaconda even makes it easy - create an environment, install your library and go. This is harder than it sounds.įor many (most?) Python codebases, running on Windows is reasonable enough. A problem I've noticed a lot of aspiring data engineers running into recently is trying to run Airflow on Windows.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |