Brief OctoMY™ update 2023

These are the changes scheduled for early to mid 2023 in the OctoMY™ project:

• Good bye Qt5 hello Qt6
• Refactoring of subsystems relating to media such as speech, video streaming etc
• Once-over of whole project
• New build system

Good bye Qt5 hello Qt6

Perhaps a little overdue, we finally support Qt6. Since we didn't really have a usable build in Qt5, we have just put that behind us and we are going all inn on Q6.

Media subsystem

This derives partly from the Qt6 transition as Qt6 introduced many changes in the underlying media related APIs and partly from the initial implementations in OctoMY™ showing it's age as many were the first parts of the codebase to be implemented and they have suffered from systemic neglect ever since.

Once-over

With a Qt version update and lots of refactoring there is always a risk of introducing lots of healthy little bugs, so this step means simply looking it over and adding tests where they are needed, running all the parts of the oprogram to see if any bugs show up.

New build system

With Qt6 came the astonishingly pragmatic (near-sighted?) decision by Qt project to move Qt away from qmake and on to CMake. The first part I completely understand, while I have learned to love qmake it still is a flawed little program and not really feasible for a large project such as Qt.

The second part of the decision; to transition to CMake feels like substituting pest for cholera. Time will tell if this was a great idea, especialkly since they had an alternative at their hands that shows a lot of rpomise as a proper next generation alternative, namely Qbs.

OctoMY™ project has some build needs that are difficult to express in either build system (being able to not maintain lists of files that are part of the project up front, and being able to dynamically chose wether a library should be linked separately or not) so we will look at adopting Qbs. So far it seems it will suit OctoMY™ quite nicely.

My pet peeves with Qt and QtCreator in 2023

I took a break from C++ and Qt for a few years as I was working on other stuff, and when I came back a unique opportunity presented itself; I could look at Qt from an outsider's perspective not being as biased as before. Well I am biased, Qt is my all time favorite platform so who am I kidding. But I have seen the world beyond and just maybe I can spot some trends where Qt moved while I was gone? This is in essence my mandatory rant on UX in programming tools, with Qt & QtCreator as a backdrop.

NOTE: I am looking at this from a community perspective. I am sure if you pay for Qt your experience will differ in some respects.

Installation

Installing Qt is still terrible, but a little better than it used to be. It still features an install wizard straight from the 1990s era, but at least it has green color to bring it into the future.

Seriously, the wasted potential is staggering. I wish someone would appoint me UX lead on the installer team, I would have a go at setting things straight. Seriously, get in touch! I will let the real-time html layout popping go I promise!

To be fair here are the options:

1. Install ancient mangled version from your favorite Linux distro's repository (not recommended)
2. Build from source.
1. Just rememer to schedule 20 minutes of your time to actually find the download link on the qt.io website. This is what happens when the marketing people takes over. I mean they really tore a page from the the old Microsoft 1990s era FUD tactic playbook here. The only contender is the delete account wizard in the Boo dating app.
2. Once you have the source.... uhm. ok so yes. I will... follow these steps and. Uhm.. It is almost as if someone made it extra hard on purpose.
3. Use some 3rd party Docker image that encapsualtes the build. Because there are no official ones. Seriously, Qt company has no official Docker images to use their shit in 2023.
4. Download online installer:
1. On linux; a payloaded bash script with a windows style install wizard inside.
2. On OSX; an img with a windows style wizard inside.
3. On Windows; a native executable with a windows style wizard inside.
4. The wizard has the worst UX of any installer I have ever used, and I have used alot of installers in my lifetime. It is actually inexcusable.
5. Download offline installer:
1. Nope, can't find it anymore. Is it there? Well you did a good job hiding it. PEBCAK.
6. Download prebuilt binaries in ZIP format.
1. I don't know if they even exist because all the files are so well hidden. If the actually exist then I am sorry for being a retard. PEBCAK.

Migration to Qt6

Migrating from Qt5 to Qt6 in my projects was actually not that bad. It took a day of search/replace/copy/paste for a fairly large codebase. Luckily I had lot's of tests to verify my changes but still the experience was good overal. QRegularExpression and QMedia* were the most hard to adapt.

The documentation spesific to migration was OK, but it could definitely have been much better. I would have loved to see a dropdown at the top of documentation pages where you could select the version you care about instead of fumbling around with google to find the relevant version. Also a collapsed comment on migration next to each deprecated member would be really helpful, instead of having all migration stuff collected on one big page.

Good documentation is one of the best sides of Qt so not complaining too much here!

CMake

CMake? SERIOUSLY?

I know this is a controversial topic. A lot of people like CMake, and a large minority dislike it. And I have a fairly strong opinion myself. Here is my breakdown;

I think that the people that like CMake like it for a completely different reason than why the other people dislike it. The people that like CMake are very pragmatic. They made it work with CMake and then it works and now we can go do something else. Or they were told by their non-technical boss to use CMake beacuse "customer X uses CMake and we must align with them".

The people that don't like CMake are more opportunistic and perfectionistic. I suspect there is a good overlap in this group with those that liked Qbs for this reason. They just can't look at CMake next to Qbs or qmake and think that CMake is the better choice. This group think (and I agree with them) while the rest of industry selected CMake, that alone is not a good enough argument to use it. Instead we should strive to surpass it on every metric beyond "popularity by mediocre middle managers" (to put it on point).

OctoMY™ is currently on a semi-hacked qmake setup that works great, but we are looking at Qbs to see if that could be even better. For OctoMY™ CMake isn't even remotely on the radar.

File types support

I don't know if this is just my memory failing me, but I definitely can't remember that there was this good support for non-Qt related filetypes like Dockerfiles, .dockerignore, *.py, *.js, *.sh, *.md and so on. After a quick highligter update they all seem to highlight beautifullty and support normal indentation. Good job!

File Operations

This peeve goes way back. Unfortunately it has worsened over time. It has several facets. I will start with the most annoying and obvious for me.

QtCreator is an IDE, which stands for Integrated Development Environment. Why is it called that? Because in olden times we used to have a development cycle like this: 

1. Open source in editor, make changes and save
2. Open terminal, invoke build command
3. Open debugger invoke program, debug, find error
4. Go back to editor make changes
5. Repeat

And conversely, the IDE will allow you to do all this from within one program, thus being "integrated".

QtCreator delivers wonderfully on this workflow with only minor annoyances, so well done!

However the IDE also must support other workflows that are very common in software development. Many of them are file related, for example:

-- As a software developer, I want to add an existing file inside my project folder to the project

or 

-- As a software developer, I want to move a file that is in a subfolder of my project into another folder

If you take ANYTHING away from this blog post, please let it be this next sentence: There really are a ton of small usecases in a developer tool that are deceivingly easy to ignore that will make or break the very essence of that tool.

Does it have shiny beautiful icons? Does it load and save projects and files blistering fast? Does it look good on all supported platforms? All good and well. But does it support the bread and butter daily file operations of a seasoned developer?

We don't want another shiny wizard. We want keyboard shortcuts, tricks and tips that work on a low level. Stuff that bear witness that the tool was made by ONE OF US. Example, it used to be that you could just rename a file in the tree view in QtCreator and use relative paths like so:

"../newsubfolder/myfile.cpp" and it would magically understand what you meant; "move myfile.cpp from it's current location  to newsubfolder". No need to bring up a menu, a bloated wizard, an external commandline or file manager.

This disappeared at some point for unknown reason. And of course being able to just move the file with the mouse like normal human being is too much to ask so I won't ask it.

Mandatory rant on UX in programming tools

If you read any good book on UX (like Don't make me think) it will certainly have a bias towards mass market appeal. What do I mean? I mean that it is part of UX syllabus and training that you should follow what the average user thinks and focus on that exclusively, remove everything else (yes actively removing it to avoid detraction of attention which is bad in UX).

If you hire a hard working, well meaning, highly educated UX person straight from school with excellent grades she may very well demolish a good software developer tool because she will in the best case misread or worst case underestimate the users of that tool.

In an IDE like QtCreator this is dangerous because we, the users, are a really smart bunch. The "average" C++ developer is not the one that benefits the company the most, it is the right side of the bell curve that will. Sure developer culture has transitions from being hero base to team based, but we all know a person or two that just are one step above the rest. And the tools should NOT deter those team members!

The sad part is that often it is the littel things, like the neat file-system hacks in the tree editor that nobody knows about. But JP the robot knows..

Media support

In general the refactoring of media support in Qt6 seems level headed. However there is one use case that has not been addressed and it is super annoying:

-- As a software developer, I want to convert a sequence of QImage into a video stream using only Qt (no external dependencies)

It is understood that this will be limited by the media codecs available on the platform, and maybe not be available on all platforms. But it is in my opinion essential for Qt to support this basic usecase.

We are not all Steven Spielberg that crave the best 8k full format presentations. We just want to chuck these QImages into a video file and save it on youtube before calling it a day. It is 2023 after all. I don't want to werstle ffmpeg or libav into my project, that is why I use Qt, so I DON'T have to deal with platform spesific stuff. The annoying part is that Qt is so close! It already integration with the native media frameworks, so why can't they just add that 200 lines to support this out of the box?

I am not alone thinking about this, my question about encoding video from a sequence of QImages from 6 years ago has 63 votes and still no resolution.

Other annoyances

Finally a list of minor anoyances in QtCreator that I felt needed to come of my chest after years of brooding.

1. The refresh action is missing in the file tree so now I find myself closing and reopeng the program every time a file has changed outside QtCreator. Luckily closing and opening QtCreatos is now super fast for some reason. It feels really snappy, so good job on that at least...
2. The swap between source and header key combination is not <ctrl>+<tab> by default. C'mon guys.
3. There is no key combination for switching between design view and editor view, and for some inexplicable reason, I am always moved to look at my design in read-only XML. dafuq?
4. The editor view for design is READ ONLY? I have to open an external text editor to edit the XML for my design? SERIOUSLY? Why don't you just remove editor view entirely, and make it a special operation to edit for those rare cases when that is even remotely relevant? Am I the only one who is annoyed by this?
5. Design view still takes up the whole screen. On modern 4k displays this is a travesty, a UX abomination. The Designer editor should really present inside a normal editor view.
6. There is no "collapse all" and "uncollapse all" buttons in tree view. I have to dig thorugh a submenu. I might as well collaps and uncollaps the tree manually and it will be faster than finding those actions *sigh*.
7. There are still issues with scaling for high DPI screens. The issues view cannot be zoomed like the other views(??) and is sometimes displaying text in way too small font.
8. The "split side by side" is wonderful, however moving the separator is not intuitive. Expected is that "everything stays the same except the size of the two views next to the separator will change in size. Actual is "depending on which separator I move, SOME windows will stretch and other not". It should be super easy to fix, just remove the separator widget in place of a horizontal layout and add widgets to take mouse dragging to change the layout. Why is this not fixed ages ago?
9. It is sometimes desirable to view files that are NOT in your project in the tree view. Yes I know, radical thought. How about adding the option to render all files found in directory with the files that are not part of the project grayed slightly out? You could even add an option to "include in proejct". Radical I know.

This concludes my rant, I hope that despite being harsh at times, it shines through how very much I love Qt. I think there is truth in the Norwegian saying "man tukter den man elsker" literally "You hurt those you love". I  yearn for it to be event better!

One year without a single update can't be good right? Happy new year 2022!

 

OctoMY has pretty much been in hibernation through 2021, even more so than in 2021. Please rest assured that we are still on track with the original plan of securing funding for the project going forward.

I still can't disclose what exactly that means, and for now it is enough to say that really it is not important. The important however thing is that the project will be revamped and brought to the front again hopefully starting this year!

2 Years is eons in tech, and as expected a lot has happened. Qt6 has been released. Robots are everywhere and the need for OctoMY is growing by the minute!

Stay tuned!

Funding of OctoMY™ and general status

Progress on OctoMY™ has as promised slowed down in year 2020. A lot has happened behind the scenes. and in this post I hope to better explain what has been going on.

In one sentence: I put development of OctoMY™ on hold temporarily while working on a project that can fund the future development of OctoMY™.

Why did I do this? I realized that organically growing the OctoMY™ project as an open source project as I first planned was not feasible from it's current state. I would have to devote all my time selling the concept to unsuspecting developers rather than advancing the development status and even then I would risk attracting very few and/or very junior resources to the project.

So I decided to switch gears and start another secret unrelated project that could generate some revenue. I will not disclose any details about this project because it is truely not relevant (and I do not wish to attract any attention to this project as it is a strictly commercial B2B project).

However after exactly one year of focusing 100% of my development time into this new project (Let's call this project "FK")  I feel at least I owe blog readers a status update!

The status is that I have gone through development hell on FK, and I am finally emerging victorious on the other end. I created an MVP over 6 months ago and was ready to launch, but decided to create a second MVP (biggest nono in history of IT) that had major feature creep. We live and we learn. The important positive takeaways here are:

• The FK project is 95% ready to launch.
• The FK that will release now is much better than the original MVP, making it much more likely to succeed.
• The FK codebase is reusable, modular and useful beyond the hot mess it was at the start. In fact many of the useful parts have now been made open source and rebranded as octomy projects in gitlab.
• I learned a lot of new technologies during my one year of FK development; Kubernetes & Docker, Python, FastAPI, flask, ... The list goes on and on... 
• I also had the oportunity to form some opinions on good practices along the way, which I would have had to do anyway for the OctoMY™ server side. I just practiced it on the FK project instead of having to do it for OctoMY™ directly.
• And maybe most importantly: When the FK project launches I hope to see a solid revenue stream that eventually will trickle into OctoMY™ in the form of my spare time (I will be able to start working on FK and OctoMY™ full time instead of just my spare time.

I hope this status update is sufficient to explain why it appears that development stopped completely. If you look in the octomy group in gitlab you will see that is far from the case. 7 (!!) new projects were added 2 weeks ago.

And with that I wish you all a happy 2021!

Deploying python pacakges to PyPi using gitlab and twine

While working on the web-minify project, I had to figure out just how it is possible to deploy a Python package to PyPi from gitlab. Here are my findings, hopefully it is useful for someone else!

Key information

• The gitlab docker executor will run each job in a separate container.
• No data is shared between jobs by default, so you have to use build artifacts to facilitate sharing between jobs
  
• The build job can thus prepare the package data in the dist/ folder ready for deployment and then mark the dist/ folder as a build artifact
  
• The deploy job will then have the dist/ folder available (all subsequent jobs after a build artifact has been defined will have access to the artifacts)
• The deploy job can then invoke twine to upload the package to PyPi
• Twine takes a username and a password from a configuration file
  
•  To avoid storing the password in the source code, a TWINE_PASSWORD environment variable is set in gitlab configuration of the project.
• PyPi supports uploading packages using API tokens instead of requiring a username/password. In this mode the username is set to "__token__" and the password is actually a long token that you get from twine config upon creation of the token instead of an actual personal password. This allows us to have some granularity of permissions, creating a token that can only access one project instead of all the projects of a user. Anyway, the token is set as a gitlab variable called TWINE_PASSWORD.

You can look at the .gitlab-ci.yaml and Makefile of the  the web-minify project to see an example of how this is done.

Good luck!