Goodbye, QML

I’m “abandoning ship” and hopping on Xamarin/Microsoft bandwagon on creating mobile apps with C#. It pains me to say that getting things to work requires a lot of effort with QML. I feel like there’s so many, many things it can do yet it tries to be so many things. At the end of the day, I was put against these options:

  • To deliver applications rapidly and gush out iterations as I go
  • The need of high-level language as C++ can be a little tedious to do some things. One can easily run into type issues.
  • Less time working on C++ backend, less time trying to understand why certain QML elements consumes over 800MB in memory (hello TextArea, you sure can’t handle 10 high-res images).

Overall, QML and C++ combo, as powerful as it is has too many memory usage issues, a steep learning curve.

Today I went over Xamarin stuff, it was relative easy to get started. I noticed that I could finally work with Activities, unlike Qt, which you have to spend a lot of time learning the android/qt cycle.

It pains me, but as most of you I also want to put my application out there in the wild as “soon as possible”. With Qt/QML I completed one mission, and that was learning C++. I did, maybe not to an expert level, but I grasped plenty.

Reducing complexity

I feel awful, yet happy, yet tired.

A bit of an annoyance is when you spend hours doing something, then a week or two later you see a new library that does that and simplifies the code greatly; allowing maintenance and readability at the same time.

So, take it as you may but if you are using classes like QXmlStreamReader or QDomDocument do yourself a favor and try pugixml library. It’s said to be faster than the previously mentioned. In terms of code readability QDomDocument is the nearest that matches pugixml due to its DOM api, yet it’s also slower and consumes a hefty amount of memory in exchange, where as pugixml stays lean.

The inadequacy of HTML, delving into C, and pondering remote jobs

I’ve been working with HTML in low-level languages like C/C++ and I must say that it’s been a rather frustrating experience. One would think that the task I’m doing should have been done ages ago, since the conception of the project. Alas, that’s not quite right. What you fail to realize is that you should NEVER underestimate the dirtiness of HTML outputs all around the world. This whole week has been about cleaning/scrubbing/sanitizing HTML. I ended up grabbing to rather awesome third party libraries, one called pugixml and the other libtidy (also known as tidy-html5).

libtidy, being a pure C library took me a while to get the hang of as I’m not that experienced in C. Even though I’m writing C++ it doesn’t mean I’m working with pointers the majority of time (mostly on the stack). It was more of a game of “copy this chunk of memory and put it as an argument in this C++-powered method so I have more control over it with the classes I’ve built. But I needed it to work, regardless on how long it took to get right because without tidy it would be really, really hard to do any scrubbing. pugixml and other XML parsers can get quite cranky while parsing misplaced tags.

Which brings me to pugixml!

Apparently QtXml module is no longer maintained as it has reached a matured state. The array of classes QDomDocument have in general feels suiting to do the job, sadly the documentation stresses that it uses too much memory, and apparently “pugixml is faster than QXmlStreamReader”

In the end. I chose pugixml because it’s just simple, it doesn’t have the annoyances QXmlStreamReader brings.

Another subject I wanted to bring is the interest of the C language.

I’ve been thinking of learning more about C. At least the basics to defend myself from cases like libtidy. I also wanted to learn more about C due to Gtk3 and due to getting closer to system programming (not that C++ isn’t capable). Mix this interest with me wanting to work with microcomputers or embedded systems and I might make something out of it.

Lastly there’s that thought of me wanting to get a job related to my field. I might try to get a job remotely, well, time will tell. For now I’m too tired, mentally spent, yet as I finish this post I have a class to attend to in college. ­čÖü

SQLite and the importance of indexes

I didn’t spend too much time on this issue, but as someone who knows the basics to intermediate in SQL it still caught me off-guard on how easy it is to have insanely slow query performances.

Take an innocent query like this. Once would think it has some sort of meta information stored already, but it doesn’t

select mytable.*, 
   (select count(*) 
    from mytable2 
    where mytable2.id=mytable.id ) as count
 from some_table;

What it does is that it will get all the columns in mytable and additionally it performs a COUNT based on id in a subquery thus the result is the total count per ID.

It took 7662ms to perform the query on two tables that have 387, and 4,800+ rows respectively. That’s not acceptable in any way, putting that sort of query in production will make your customers leave eventually due to the slowness of the system.

I googled a bit on the issue and found that sqlite doesn’t store any meta or index. I went ahead and create an index based on the second table, as the first one already has an index (primary key).

I managed to trim those seconds to 240ms, which imho is pretty acceptable, yet makes me feel uneasy.

The result?

select mytable.*, 
   (select count(ROWID)
    from mytable2 
    indexed by my_awesome_index
    where mytable2.id=mytable.id ) as count
 from some_table;

That’s all it took.

I’m still not completely satisfied with results like this, I will continue finding a way to trim those 200ms away to 20 or 50. I have an idea on how to do it but for now this will do.

Another take at const-correctness and “optimizing” code.

Most of what I did today was cleaning code… or refactoring code. I wasn’t happy with the current codebase as in there were a lot of copies of strings being made instead of simply making references and that’s what made it so unsatisfying as I was duplicating codes of data.

One of the aspects of my application is that most of the time I don’t need to modify data thus using const & comes handy most of the time plus it helps the compiler as much as it helps me if I make a wrong assignment if(cookies = cream). It took me like 2 hours to fully refactor the codebase … I wouldn’t even call it refactoring at this stage as nothing needed to be moved away.

After I was done with the task I sat down, scratching my head wondering why do I have so many unneeded objects in the heap. Well, the wonders of both C and C++ is having the ability to write your allocators and how you literally manage memory. Objects in the heap–mostly created by new–can be harder to keep track and free in a large codebase, thankfully shared pointers, scoped pointers have been introduced, and yet, people like me still use naked pointers–sorry, at least give me points for doing my homework.

Interestingly while I’d dare say with my naked eye I’ll say that the app launches faster. However Qt isn’t that forgiving as having QML Engine running can cost you 40-50mb of ram, just starting. It’s actually a tiny price to pay compared to what you get with Qt in exchange. I’m sure I could lower the memory usage but… who knows…

Either way… I must say I’m proud of myself, with the progress I have done so far. To the mere veteran, or armchair programmer it may not mean much; but to me means a lot as I have stayed truth to my goal of learning C++.

Onward, I say!

Writing this as a relief and to vent a bit… I guess.

So first of all Happy New Year! Hopefully this year will be filled with pure awesomeness. Secondly, my god managed network requests can really burn you.

I stumbled upon two serious bugs, one being that Atom reader wasn’t doing its job properly and I ended up writing it completely thus now it shouldn’t have any problems. Now, parsing through content can be sometimes tedious but manageable. I have done it so many times in the past in unrelated projects that I more or less know how to handle certain issues.

Networking sadly… is a new issue for me, but see my conclusion wasn’t that the tool was bad… rather that my application’s design is not bad but doesn’t really adjust to how QNetworkAccessManager plays. The designed classes to read Atom and make requests were both really bad designs as in, at least in the network part, was naively crafted. I had this great idea of stress testing my application to see how it would handle large amount of requests, in turn I had to implement around 300 line of codes to fix a few annoying issues and thus took care of:

  • Web servers don’t play nicely, don’t expect it to accept your request if your request don’t have headers like Accept, Accept-Encoding, User-Agent. I then by mistake added gzip in my Accept-Encoding which forced me to remove it as I currently have no interest in reading compressed content (yet).
  • No timeout. No timeout is just that. If your request doesn’t make it don’t expect the reply to erase itself. I didn’t have any timeout management
  • No supervision on how many requests to make per… … per what? there wasn’t any restriction thus it made the QNetworkAccessManager behave completely different.

The solution I have created felt so hack-ish I’m not slightly happy about it thus I’ll have to redesign how the network part interacts and how the network part interacts with the database.

While I was testing I ran into another problem: there’s no way to prioritize signals in Qt so I have to do the abstraction myself, which I don’t mind. To me providing a responsive interface is more important than the availability of data (which should fall into a “normal” priority). So in that department … I literally have to refactor a large portion of the application which is scary itself.

But… I must cut my losses and carry on. Most of the problems have been bad design itself… which is good and bad. Good because most of the logic is sound, bad because the design is a clusterfuck to manage.

Aight, well… go me, I guess.

6 months ago I started this personal project

It feels like forever.

Even if it’s all I talk about in this blog I have I decided to keep going with this as if I kept pondering whether to do it or not I would have never been at this stage.

2015-12-25

I refactored and added plenty of code this week. Refactoring because let’s be honest when I’m faced with a new framework and a new language I’m bound to code little building blocks of what we call “experiments”, the very nature of not knowing pushes you to make spaghetti code in some aspect but you must be always willing to part ways with your old code and refactor as much as you can.

View post on imgur.com

Yes, yes. Give into the “Yes!” chants. (Daniel Bryan if in case you aren’t into the whole wrestling thing).

After refactoring, implementing new code, and deleting old code it feels oh so much better than before to the point that I feel pretty satisfied of my work so far in the app. I think what I’ve learned throughout the whole journey of this will be invaluable in the future if you consider the investment in C++ and Qt worth it that is, which to me it is.

With the changes I’ve done I can finally concentrate on the interactions between the database backend and the models involved so it’s always synchronized.

In a related note, I must say I’m impressed by the blazing speed of SQLite/Qt in general. I still have a few worries about it but… so far it’s been a pretty darn straightforward easy to use abstraction.

Thing is I’m doing my best to keep a good pace and finally finish it. I hope all my efforts eventually pays off, if not… I won’t really call it a failure as I gained so much in the process.

Time will tell..

Implementing TreeView expandAll/collapseAll starting on top-levels

I should start by a disclaimer that I’m neither a C++ or Qt expert, there may be better or elegant ways to do it and I have found this implementation is more straightforward than digging in TreeView’s QML source code.

So we have a small problem, the TreeView in QML doesn’t provide any way to expand or collapse items. Alright, what would be the best way to tackle this? Let’s look at the available options at our hands:

  • TreeView QML provides the following methods:
    • expand (QModelIndex)
    • collapse (…)
    • others …

And then we have implemented the AbstractItemModel. If you sit for a moment and think we can actually aggregate new methods to our AbstractItemModel with Q_INVOKABLE and call them in our QML code.

So we will implement a new method called getChildrenIndexes (that of course is sort of a bad name when I think about it).

QVariantList GenericModel::getChildrenIndexes()
{
    QVariantList indexes;
    GenericNode *parent = rootItem->child(0)->parent();

    for(int i = 0; i != parent->childCount(); ++i) {
        GenericNode *child = parent->child(i);
        indexes.push_back(createIndex(i,0, reinterpret_cast<quintptr>(child)));
    }

    //reinterpret_cast<quintptr>(c) ( found in inners of QModelIndex)



    return indexes;

}

Then put the Q_INVOKABLE QVariantList getChildrenIndexes(); in your model header.

In our QML code it will go like this:

                onAssignSubscriptionModel: {
                    treeView.model = model; // TODO: throw a WorkerScript at it. 
                    var someIndexes = treeView.model.getChildrenIndexes();
                    console.log(someIndexes); // Our QVariantList gets converted to an array
                    for(var i = 0; i <= someIndexes.length - 1; i++) {
                        treeView.expand(someIndexes[i]);
                        //console.log(treeView.isExpanded(itemIndex));
                    }
                    someIndexes = undefined; // not sure if effective but I want it garbage collected afterwards. 

                }

After that? Your items should be automatically expanded. Note that this is an expensive process, it’s not much about creating the indexes but more that it will tax the QML engine side if you have a lot of items. At the same time I’d like to say that I did go through it with around 13k items in my database and didn’t notice any slow down, keep your eyes open though.

A quick summary? We got the indexes of our top-level items. Do remember that the root item is always invisible. Which brings me to

remember to adapt this to whatever node structure or container you have nobody should expect rootItem->child(0)->parent(); line in anyone’s AbstractItemModel meaning, you probably don’t have a child or parent method, maybe.

Delving into Qt Quick/QML, states and transitions

My apologies if the video is a big too long in terms of height. I’ve been working really hard to get my app out there, yet the more I move closer to my goals the more there is to do. I’m not sad about that though, if anything this has been one hell of a productive week.

C++/QML interaction

I feel like I should start talking about the interactions between Qt Quick/QML types with your C++ backend. I must say that I’ve had the wrong idea since the beginning. Why? Well, if you must ask I will tell you a nightmare prone story where my entry CPP file began with searching QObjects from the QML engine to establish a connection between my backend and the visual aspect(types) of my app.

At first sight it doesn’t sound bad, but it is. When you use QtQuick/QML (referred from here on as QML) the main idea is to provide a rich dynamic interface and have it all typed up with almost no effort. Yet, one of the questions you always ask yourself is “how to interact with the backend” and in the documentation it explicitly starts talking about finding children in the QML engine root object so you can connect them with your backend.

Depending on the type of your app there’s one huge problem: What if my app is too dynamic and the types/elements of it are always getting destroyed and recreated due to wanting to free memory?

The answer lies within either making the root object a hub of communication OR extending QML with new providers type. If you have read the documentation at some point you will come across an example that starts with creating a QML Type called Message and that message will provide and contain said message through its usage. After you are done designing the properties and members of Message object in your CPP backend you can register it with qmlRegisterType. A bit more info here, to those who are interested.

States

I have found that I have more control making my qml types have states or certain properties that hints the internals of my app of when to do certain things. States in QML helps a lot in that, while it’s commonly used in transition I can also see it as a semaphore to control the flow of how you want to direct the user.

As you saw in the video at the start of this post you can see I’m also using states.

Nothing is certain

Right now there’s just no right way to write your qml files. There’s no “best practices” excluding the memory management part in the documentation as it’s exclusively for memory management. How you lay out the visual objects and how you communicate and bridge it with your backend is up to you. I don’t really know if this is really good or bad, you just have to come up with an “efficient” way of not compromising the performance of your app.

All in all, it’s been a pretty nice experience. I just wish the documentation improved.

Bugs, bugs everywhere!

You know one of the most excruciating experience is when you are new to something the first thoughts you will have after analyzing an specification is:

  • Ok, I got this. All I need is this, this and that.

In web development, to me, it became something like that. I already knew what to do, how to do it and how to put the pieces together. Now with C++/Qt everyday is a challenge. Most stems from the lack of familiarization of Qt Framework API, and C++ standard library. Everything is so alien, yet so exciting at the same time, and so, so exhausting since you burn yourself a bit in the process.

Sometimes I like to take some time off and explore other languages, other frameworks/platforms. Recently I’ve found a need to learn C# and Visual C++ ^^. I will eventually learn them, my struggle begins with designing the application… and I’m guilty of over-complicating this task. Why? I have to keep repeating “we can always refactor it later”.

I guess, I’m just overwhelmed by all these things. I have worked and invested so much time, sometimes I have to wonder if it’ll be enough to get hired in a decent software development workshop. Hopefully next year I’ll find out after I graduate, but for now I must keep learning as much as I can and put my knowledge in practice.

As for my app? Decent progress, I’ve done a lot of bug fixes mostly related to the XML parser as for some reason Qt’s XML stream parser can drive you nuts. There’s still a lot to do, yet, the backend is more or less shaping up to be stable enough and the data from the parser has been inserting with no problems, meaning once I get the backend stabilized all I have to do is finish the frontend then the testing begins.