Development Notes #5 Getting organized a bit more for execution

Over the three weeks there’s been a lot of questions on my mind on how to execute and do the presentation on some things. These things, they take time, because first impression is at best one of the first thing I have to focus on.

If there’s one thing I could say is very taxing is the user experience. The first impressions, the never-ending testing of things and the sheer insecurity and impostor syndrome that develops along this journey. I want my project to be successful, like anybody else. Sometimes it drives you a bit crazy on I hope I hope that it’s a hit around the world because who doesn’t want to slack off a little bit and get some vacation. Of course, technically speaking we can’t…. really do much about vacation with the COVID-19.

Truth is, developing a product is itself a gamble. No matter how much you study you can’t quite penetrate the market as you would hope because either the presentation is done poorly or the execution of the project is done poorly.

Documentation and presentation are keys that I have identified. I need to prepare a lot of documentation or a modest amount of documentation for the people to develop software.

I honestly don’t know how things are gonna go. But, we’ll see in the future. For now I’m just continuing working on this and hoping for the best.

Today I was able to finalize the environment for development on my new laptop. This wasn’t really a slow thing to do but bringing my laptop up to date and preparing everything did take a while… or most of my evening.

Now that everything is ready I think I can finally proceed on what to do next. There’s been a lot of things that I’ve been setting up in the background… working slowly through the items.

In a weird sense… I did complete a major portion of it. Now I’m just making sure that the following items are represented well enough.

Now, I know I haven’t said what I’m working on. It’s too abstract to understand at the moment but eventually I’ll be able to talk more about it.

SQLite and the importance of indexes

I didn’t spend too much time on this issue, but as someone who knows the basics to intermediate in SQL it still caught me off-guard on how easy it is to have insanely slow query performances.

Take an innocent query like this. Once would think it has some sort of meta information stored already, but it doesn’t

[sql]
select mytable.*,
(select count(*)
from mytable2
where mytable2.id=mytable.id ) as count
from some_table;
[/sql]

What it does is that it will get all the columns in mytable and additionally it performs a COUNT based on id in a subquery thus the result is the total count per ID.

It took 7662ms to perform the query on two tables that have 387, and 4,800+ rows respectively. That’s not acceptable in any way, putting that sort of query in production will make your customers leave eventually due to the slowness of the system.

I googled a bit on the issue and found that sqlite doesn’t store any meta or index. I went ahead and create an index based on the second table, as the first one already has an index (primary key).

I managed to trim those seconds to 240ms, which imho is pretty acceptable, yet makes me feel uneasy.

The result?

[sql]
select mytable.*,
(select count(ROWID)
from mytable2
indexed by my_awesome_index
where mytable2.id=mytable.id ) as count
from some_table;
[/sql]

That’s all it took.

I’m still not completely satisfied with results like this, I will continue finding a way to trim those 200ms away to 20 or 50. I have an idea on how to do it but for now this will do.

Pelican2Ghost plug-in released

Pelican2Ghost is a plugin for Pelican site static generator that generates a JSON file so you can import it in Ghost blogging platform.

How to use?
git clone git@github.com:allenskd/pelican2ghost.git

Open your pelicanconf.py and add

PLUGIN_PATHS = ['/my/base/path']
PLUGINS = ['pelican2ghost']

For more information please check pelican2ghost Github

I found a bug!

Please do not comment that you found a bug. I’d appreciate it if you could report it, here.