Everyone posting their pip/build/runtime errors is everything that's wrong with tooling built on top of python and its ecosystem.
It would be nice to see the ML community move on to something that's actually easily reproducible and buildable without "oh install this version of conda", "run pip install for this package", "edit this line in this python script".
Interesting question to me is whether it's actually in part fundamental to the success of Python. There are plenty of "clean" ecosystems that avoid at least some of these issues. But in generally they fail to thrive in these type of spaces and people keep coming back to ecosystems that are messier.
Is it possible that creativity and innovation actually require a level of chaos to succeed? Or alternatively, that chaos is an inevitable byproduct of creativity and innovation, and any ecosystem where these are heavily frowned on deters the type of people who actually drive the cutting edge forward?
Just putting it out there as food for thought. It does translate back to quite often, that it acutally does make a lot of sense to do your prototyping and experimenting in one ecosystem but then leave that behind to deploy your production workloads where possible.
It's not just the python, this is the experience practically everywhere and that's why people create containers etc. It's excruciatingly hard to setup the environment to start doing anything productive these days, you can't just start coding unless you use an IDE like Xcode or PyCharm.
That's probably fair. I've found pyenv/venv/poetry/pipenv/pip/conda/etc more frustrating than ESM/CJS/yarn/npm/PNP/etc, but then I just do a lot less Python stuff than JS these days.
Maybe the difference is the legacy and the community? Golang and Rust kind of languages are geared toward software engineers and are pretty new, meanwhile things like Python or JS or anything very popular are used as tools by people great at things that are not necessarily in the domain of software engineering but they created very useful libraries and tools regardless which usually means piles and layers of very useful code that has poor engineering.
I don't think this is particularly fair. This is literally hours old, and people installing now are really debugging rather than installing a "finished" build.
The forks are weird mashups of bits of repos, and running on a M1 GPU is something that barely works itself.
Give it maybe 3 months and it will be much smoother.
I think it is fair, actually. This is not unique to hours-old python projects, it's a common theme among almost all python tools I've used.
I have a suspicion that that something written in Julia, Go, Rust, or possibly even C wouldn't have nearly this many issues. I'm not talking about debugging the actual functionality of the software, but rather the environment and tooling surrounding the language and software built with it.
This project in particular should be an easy case because you know the hardware you'll be running on ahead of time.
I'm ranting a bit, but I've tried so many tools based on python and almost _none_ of them built/installed/ran correctly on the happy path laid out in each project's readme.
I do think that experience helps here. I have a recipe for installing Python that works on most python projects most of the time.
git clone <project>
python3 -m venv ./venv
source ./venv/bin/activate
pip install -r requirements.txt
deactiviate # need to do this to include the correct command line tools in path (eg Jupyter)
source ./venv/bin/activate
Done.
On a Linux or Intel Mac system this works with pretty much every reasonable Python project.
On M1 Macs the situation isn't great at the moment, though.
As if compiling any given C program wasn't also a crap shoot.
People also tend to forget that these ML packages are ridiculously complicated, and have a lot of dependencies not just on other libraries but on particularities of your system.
That, and ML researchers can't also be expected to be good at everything. They are busy doing ML research and waiting to be given a recipe to follow.
Meanwhile I can put together a Python package in my sleep that works perfectly on pretty much any system, but I don't know a damn thing about Autotools and would probably make a total mess if I tried use it. Or CMake. Or whatever Java uses.
This "look Python bad!!" stuff has some merit (if only historical), but it mostly amounts to FUD and does a big disservice to the people who have worked hard over the past few years to get everything fixed up.
Plus at least one of the issues I see here is because people disregarded the instructions and used Python 3.9 even though it says to use 3.10 because the code assumes it's running under 3.10.
Also, sorry to double post, but you don't actually need to activate the venv. You can just invoke `venv/bin/pip`. That should hopefully save you a bit of time and annoyance.
Firstly, non-experiences Python users (the kind who need recipes to follow) will also blindly follow installation instructions on websites. These all say "pip install xxx" and users will inevitably forget they need to change the path to make that work for their environment. By getting into the habit of activating the environment this problem goes away.
Secondly, you do need to activate the venv so that command line tools in ./venv/bin are used (instead of whatever is in your default path).
This includes both the correct version of python (if you set it when you setup the virtual environment) and importantly jupyter. From personal experience there's a whole set of pain when you don't realize your jupyter isn't the one inside the virtual environment and so is using other libraries.
Because many shells cache the location of executables, if you don't deactivate and reactivate your virtual environment after installing Jupyter into it then it may (in some circumstances) use a globally installed version.
My anecdotal experience is that it briefly gets better, then much worse. Build scripts downloading tarballs from dead URLs and dependencies on unspecified versions of libraries that have since had breaking API changes are frequent issues.
…it’s still awesome that people put in the effort to do these things at all, but the tools often have a tendency to make me feel like an archaeologist trying to piece what ancient artifacts are missing and how they were supposed to all fit together.
I ran into many of these same problems when trying to replicate ROS environments five years ago. It's not Python's fault, it's just academics being bad at releasing software. After all, why would they?
In a university, ten steps to replicate the PI's personal environment is perfectly fine. How would releasing a single binary make their life any easier? Why would they bother?
I found a pretty great Docker image for the SD webui. I was forced to extreme measures since NixOS isn't super friendly with Conda (and I was particularly lazy).
Looking through the top level comments, there are only 2 like that and they are about build errors in dependent native libraries written in other languages, which is not really the Python ecosystem's fault (as the author has chosen not to distribute prebuilt stuff).
There are working CUDA Docker images that work both on Windows and Linux for stable diffusion so the package including dependencies already exists. It's just they the standard packaging method doesn't work well on Apple hardware.
Just don't mess up your system. The tutorial linked works perfectly fine in a clean python environment.
That XKCD is more about messing up your system by not knowing what you're doing but randomly following shitty tutorials which suggest stuff that collide with each other... Python's only fault in this is that it's a simple language and thus attracts people who aren't software engineers (students and math majors) and thus mostly don't know or care how to keep your system clean but love writing tutorials.
It's pretty easy to keep your pythons clean, don't use conda, never run pip with sudo, never run pip with --user..., never run pip outside a virtualenv (a good safety measure for that is to have pip point to nothing in your user shell, you can access system python with python3/pip3 if needed)... To check your python is clean, create a new environment and run pip freeze, it should output nothing.
pyenv is a non-destructive system for managing multiple pythons and virtualenvs (all pythons and envs get installed into ~/.pyenv), pip is a good system for distributing dependencies (when library authors don't skip out on providing binary wheels, and software authors use pip freeze to generate requirements.txt files).
If your solution is “spend a lot of time learning a lot of rules and make sure you never accidentally do something you shouldn’t” then your solution is fragile and shitty.
You know what’s radically simpler? Shipping your damn dependencies so all any user needs to do is double-click the launcher and it will always work no matter what weird and bad things your user as done. The fact that “keep your system clean” is a thing anyone thinks about ever is a catastrophic failure.
stable diffusion is a work of researchers, not engineers, it's not a released product, it's an attachment to a research paper... researchers shouldn't need to know anything about releasing a software product...
you see it here on hacker news, because it's open source and thus hackers can and do already play with it..
as it's an interesting piece of software, I'm pretty sure someone will eventually make an easy to use GUI app based on it that ships all dependencies, but it's not yet at that stage,
this article is literally about the first easy (for a software engineer who knows python) way to run it on M1 macs, these people literally just discovered which versions of their dependencies work and share their work to invite other people to help them build something on top of it....
It would be nice to see the ML community move on to something that's actually easily reproducible and buildable without "oh install this version of conda", "run pip install for this package", "edit this line in this python script".