that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
#Awsglue
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product
The first experience a developer should have with #spark should not be counting words. Instead, it the product should just punch the developer in the face, over and over.
`Select * From oracle table` should be so far out of reach that should a developer ever succeed, then counting words will feel like a miracle.
♡ 2 ↻ 0AWS Glue
- No VPC/Subnet for compute
- Unless you use "connnectors" (opaque java/jdbc in docker things)
- python log to info? ok. log to debug/warn? Kaboom, opaque message from java
- Where are the logs? Oh, like 10+ places to look, sometimes cloudwatch, sometimes s3
- you can install libs with pip, but not oracle stuff, that requires arm64 binaries that don't exist
- You have data frames! But they are eager evaluated and select the whole table first. :(♡ 0 ↻ 0- 100s of databases supported! 10s of pages of documentation, or about 6 lines of documentation for your particular data source.
- What to use ray or pyspark? Well, the choice is easy, because no matter which one you choose, the experience is just pain- no VPC, no subnets, it's arm64, logs and error logs are splatter all over the place♡ 0 ↻ 0
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.