Coffee Space


Listen:

Akregator RAM Usage

Preview Image

Preview Image

NOTE: This article is currently a work in progress. It may not have an interesting resolution, but you get to see the process in action.

Background

I used to use Thunderbird for my RSS feeds, but a while back it entirely corrupted my feed and I had to put in a lot of effort to recover my subscriptions. I managed to get back to an OPML file and simply decided in the end to move to a different feed reader.

After looking around, I found akregator seeming to be a simple, old and well supported feed reader. From what I can tell it compiles down to a binary, rather than being one of these interpreted language programs that one day decides it doesn't want to work, because a Node package that does some simple task is suddenly deleted.

Akregator has some problems, like not being able to switch off the tabs system (I only want to download content or view it in a browser), supporting Javascript (my feed reader shouldn't have a Javascript engine, that's what my browser is for) and when it crashes it forgets what you have already read (which seems pretty dumb).

That said, it generally has been more reliably than Thunderbird so far, which probably is a pretty low bar to pass - given it corrupted my feeds and silently stopped updating them.

Problem

The other day my laptop running Ubuntu started freezing up and completely locked up. All software updates are disabled (including snap), which was the previous cause of instability. After running for weeks, I was surprised that it suddenly became so unstable.

I of course was forced to reboot my machine manually and was quite cautious thereafter, checking back at the System Monitor regularly. To my surprise. I see akgregator of all things consuming over 1GB of RAM! It was giving Firefox a run for its money, and my version of Firefox has over 1000 tabs open! What the hell!

Debugging (2021-08-04)

First things first, I went to DuckDuckGo and searched for similar bugs - to which I found none. This was kind of annoying, so I sat on the problem a little longer, played with the settings and the RAM usage continued to increase.

I then went to IRC on the #kde-bugs channel on the Libera server. There somebody recommends to lodge a bug report over on KDE. I check for existing bugs there to do a cursory glance to try and prevent replication of efforts. After scrolling through I find nothing.

After finding the relevant version information, I registered and submitted a report. Some details from that report:

0001 SUMMARY
0002 
0003 I am currently observing ~250MB of RAM usage increase per day. This lead to a
0004 system crash.
0005 
0006 Currently Akregator is consuming 1.7GB of RAM, 4.9GB of virtual memory.
0007 Compare this to Firefox with hundreds of active tabs, consuming 1.4GB of RAM
0008 and 8.9GB of virtual memory.
0009 
0010 STEPS TO REPRODUCE
0011 1. Large number of RSS feeds (OPML export is ~300kB and ~700 lines)
0012 2. Set checking time to 30 minute intervals
0013 3. Wait for 24 hours
0014 
0015 OBSERVED RESULT
0016 
0017 A ~250MB RAM increase over 24 hours. Over several weeks this will lead to a
0018 system lock-up/crash (in this case).
0019 
0020 EXPECTED RESULT
0021 
0022 A little to no RAM increase over 24 hours.
0023 
0024 SOFTWARE/OS VERSIONS
0025 Windows: -
0026 macOS: -
0027 Linux/KDE Plasma: Ubuntu 21.04
0028 (available in About System)
0029 KDE Plasma Version: -
0030 KDE Frameworks Version: 5.80.0
0031 Qt Version: 5.15.2
0032 
0033 ADDITIONAL INFORMATION

Then I thought to myself, perhaps there is more I can do.

After looking online at several solution for monitoring RAM usage for a given process, I settled on and adjusted an answer from StackOverflow:

0034 delay=300
0035 pid="$1"
0036 
0037 top -b -d $delay -p $pid | awk -v OFS="," '$1=="top"{ time=$3 }
0038 $1+0>0 { print time,$1,$NF,$9,$10; fflush() }'

After running for about 9 hours, we see this:

RAM usage study

RAM usage study

As you can see, RAM usage is increasing over time as I suspected. It appears to be increasing approximatly 20MB per hour and is mostly linear. There appear to be two lines, perhaps the top one is when it is actively processing new incoming RSS feeds.

Other than this, it's still too early to draw conclusions. I now continue to wait for 24 hours and see if somebody becomes interested in my ticket.

Debugging (2021-08-05)

As I suspected, things didn't get better inside the 24 hours observation time:

24H RAM usage study

24H RAM usage study

Next steps:

I have begun running valgrind using the advice from StackOverflow:

0039 valgrind --leak-check=full -v --track-origins=yes --log-file=log.txt valgrind

I did find that --track-origins=yes was exceptionally slow, so I removed this to get something like a usable experience. I will run for a few hours before closing to ensure I definitely get some data.

After testing - is there a leak? Yes. From the log we see:

0040 ==595152== LEAK SUMMARY:
0041 ==595152==    definitely lost: 298,872 bytes in 357 blocks
0042 ==595152==    indirectly lost: 2,469,366 bytes in 10,308 blocks
0043 ==595152==      possibly lost: 232,592 bytes in 476 blocks
0044 ==595152==    still reachable: 10,728,421 bytes in 104,272 blocks
0045 ==595152==                       of which reachable via heuristic:
0046 ==595152==                         newarray           : 936 bytes in 5 blocks
0047 ==595152==                         multipleinheritance: 29,552 bytes in 21 blocks
0048 ==595152==         suppressed: 0 bytes in 0 blocks
0049 ==595152== Reachable blocks (those to which a pointer was found) are not shown.
0050 ==595152== To see them, rerun with: --leak-check=full --show-leak-kinds=all
0051 ==595152==
0052 ==595152== Use --track-origins=yes to see where uninitialised values come from
0053 ==595152== ERROR SUMMARY: 14687 errors from 1145 contexts (suppressed: 0 from 0)

From what I can tell (with my very limited knowledge), we see to be leaking memory mostly from libQt and more specifically libQt5WebEngineCore:

0054 ==595152== 836,845 (186,208 direct, 650,637 indirect) bytes in 92 blocks are definitely lost in loss record 25,600 of 25,601
0055 ==595152==    at 0x4842FB3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
0056 ==595152==    by 0x744FCC9: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0057 ==595152==    by 0x77B2AB2: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0058 ==595152==    by 0x9E17A8A: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0059 ==595152==    by 0x9E184AE: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0060 ==595152==    by 0x9E1C912: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0061 ==595152==    by 0x9E1F664: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0062 ==595152==    by 0x9E14D44: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0063 ==595152==    by 0x9E1505D: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0064 ==595152==    by 0x9E32CE0: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0065 ==595152==    by 0x97700DA: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)
0066 ==595152==    by 0x9783A93: ??? (in /usr/lib/x86_64-linux-gnu/libQt5WebEngineCore.so.5.15.3)

To be sure, I would probably need to compile from source. If somebody does eventually answer the ticket, they will likely ask me to use the most up-to-date version and confirm the problem still persists there too. For now though I will wait for somebody to reply and keep logging RAM increase over time.

Debugging (2021-08-06)

After running for a while longer, the trend appears to still hold true, even from a recently started process:

24H RAM usage study

24H RAM usage study

I thought some more about building from source, and after a little searching, found Akregator on GitHub. I saw there a comment about a dedicated channel on IRC, but haven't been able to get hold of anybody there yet:

0067 IRC:  irc://irc.libera.chat/#akregator

I asked about build dependencies on IRC, but got no comment there. I decided I would just give it a go and see what happens.

First of all, we need to clone and try an initial build:

0068 cd $WORKING DIRECTORY # Got to where we want to clone it
0069 git clone git@github.com:KDE/akregator.git
0070 cd akregator/         # Go to where we clone it
0071 gitk --all            # Check everything looks good
0072 git checkout v20.12.3 # Checkout the version I want to build
0073 cmake .               # Attempt to generate the Makefile

It gives it a go, but fails:

0074 -- The C compiler identification is GNU 10.3.0
0075 -- The CXX compiler identification is GNU 10.3.0
0076 -- Detecting C compiler ABI info
0077 -- Detecting C compiler ABI info - done
0078 -- Check for working C compiler: /usr/bin/cc - skipped
0079 -- Detecting C compile features
0080 -- Detecting C compile features - done
0081 -- Detecting CXX compiler ABI info
0082 -- Detecting CXX compiler ABI info - done
0083 -- Check for working CXX compiler: /usr/bin/c++ - skipped
0084 -- Detecting CXX compile features
0085 -- Detecting CXX compile features - done
0086 CMake Error at CMakeLists.txt:8 (find_package):
0087   Could not find a package configuration file provided by "ECM" (requested
0088   version 5.75.0) with any of the following names:
0089 
0090     ECMConfig.cmake
0091     ecm-config.cmake
0092 
0093   Add the installation prefix of "ECM" to CMAKE_PREFIX_PATH or set "ECM_DIR"
0094   to a directory containing one of the above files.  If "ECM" provides a
0095   separate development package or SDK, be sure it has been installed.
0096 
0097 
0098 -- Configuring incomplete, errors occurred!
0099 See also "./CMakeFiles/CMakeOutput.log".

So here we can see it failed to build based on ECM, so we need to find out what it is complaining about and likely install something to meet that dependency. This is the list of things I ended up needed to download via sudo apt-get install <PACKAGE> (your milleage may vary):

Finally I was able to run cmake . and get a good message (the list below of REQUIRED packages may give you a better list for installation):

0100 Installing in /usr/local. Run ./akregator/prefix.sh to set the environment for akregator.
0101 -- Could NOT find KUserFeedback (missing: KUserFeedback_DIR)
0102 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
0103 -- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
0104 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
0105 -- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
0106 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR
0107 -- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
0108 -- The following OPTIONAL packages have been found:
0109 
0110  * QGpgme
0111 
0112 -- The following REQUIRED packages have been found:
0113 
0114  * ECM (required version >= 5.75.0)
0115  * Qt5Qml (required version >= 5.15.2)
0116  * Qt5QmlModels (required version >= 5.15.2)
0117  * Qt5Quick (required version >= 5.15.2)
0118  * Qt5WebChannel (required version >= 5.15.2)
0119  * Qt5Positioning (required version >= 5.15.2)
0120  * Qt5WebEngineCore (required version >= 5.15.2)
0121  * Qt5WebEngineWidgets
0122  * Qt5PrintSupport
0123  * Qt5 (required version >= 5.13.0)
0124  * Grantlee5 (required version >= 5.2)
0125  * KF5Crash (required version >= 5.75.0)
0126  * KF5DocTools (required version >= 5.75.0)
0127  * KF5KCMUtils (required version >= 5.75.0)
0128  * KF5NotifyConfig (required version >= 5.75.0)
0129  * KF5SyntaxHighlighting (required version >= 5.80.0)
0130  * KF5TextEditor (required version >= 5.75.0)
0131  * KF5Notifications (required version >= 5.75.0)
0132  * KF5Syndication (required version >= 5.75.0)
0133  * KF5Parts (required version >= 5.75.0)
0134  * KF5KontactInterface (required version >= 5.16.3)
0135  * KF5Libkdepim (required version >= 5.16.3)
0136  * KF5MessageCore (required version >= 5.16.3)
0137  * KF5MimeTreeParser (required version >= 5.16.3)
0138  * KF5MessageViewer (required version >= 5.16.3)
0139  * KF5Sonnet (required version >= 5.80.0)
0140  * KF5PimTextEdit (required version >= 5.16.3)
0141  * Qt5WebEngine (required version >= 5.13.0)
0142  * KF5WebEngineViewer (required version >= 5.16.3)
0143  * KF5AkonadiMime
0144  * KF5ItemModels (required version >= 5.75.0)
0145  * Qt5Gui (required version >= 5.13.0)
0146  * KF5Service (required version >= 5.80.0)
0147  * KF5Completion (required version >= 5.80.0)
0148  * KF5ItemViews (required version >= 5.80.0)
0149  * KF5JobWidgets (required version >= 5.80.0)
0150  * KF5Solid (required version >= 5.80.0)
0151  * KF5Auth (required version >= 5.80.0)
0152  * KF5WidgetsAddons (required version >= 5.80.0)
0153  * KF5ConfigWidgets (required version >= 5.80.0)
0154  * KF5XmlGui (required version >= 5.80.0)
0155  * KF5WindowSystem (required version >= 5.80.0)
0156  * Qt5Network (required version >= 5.14.0)
0157  * Qt5Test (required version >= 5.13.0)
0158  * Gettext
0159  * KF5Contacts (required version >= 5.75.0)
0160  * KF5GrantleeTheme (required version >= 5.16.3)
0161  * Qt5Widgets (required version >= 5.13.0)
0162  * KF5AkonadiContact (required version >= 5.16.3)
0163  * KF5CoreAddons (required version >= 5.75.0)
0164  * Qt5Core (required version >= 5.14.0)
0165  * KF5Codecs (required version >= 5.75.0)
0166  * KF5Mime (required version >= 5.16.3)
0167  * KF5IMAP (required version >= 5.16.3)
0168  * KF5PimCommonAkonadi (required version >= 5.16.3)
0169 
0170 -- The following OPTIONAL packages have not been found:
0171 
0172  * KUserFeedback (required version >= 1.0.0), User Feedback lib
0173    Allow to send Telemetry Information (optional). It can be disable in apps.
0174 
0175 -- Configuring done
0176 -- Generating done
0177 -- Build files have been written to: ./akregator

And now we can see the built binaries:

0178 $ ls bin/
0179 akregator                       akregatorstorageexporter
0180 akregator_config_advanced.so    downloadfeediconjobtest
0181 akregator_config_appearance.so  kontact_akregatorplugin.so
0182 akregator_config_archive.so     libakregatorinterfaces.so
0183 akregator_config_browser.so     libakregatorinterfaces.so.5
0184 akregator_config_general.so     libakregatorinterfaces.so.5.16.3
0185 akregator_config_plugins.so     libakregatorprivate.so
0186 akregator_mk4storage_plugin.so  libakregatorprivate.so.5
0187 akregatorpart.so                libakregatorprivate.so.5.16.3

Next we run the program and make sure the memory leak still exists in the version we just built from source:

0188 ./bin/akregator

This will take a little while, so I will follow up in a further update. The only other thing to do was update the ticket regarding my current actions and plans.

Debugging (2021-08-07)

After nearly 24 hours, we observe our RAM study on the built-from-source version:

24H RAM usage study

24H RAM usage study

Some interesting observations:

0189 ./bin/akregator: symbol lookup error: /usr/lib/x86_64-linux-gnu/qt5/plugins/akregator_config_userfeedback.so: undefined symbol: _ZN9Akregator19UserFeedBackManager4selfEv

The built version suggests the following versions (supposedly the same as before):

Not entirely sure where the different is coming from here... Time to update the ticket!

Debugging (2021-08-08)

I recieved a comment on the ticket that suggests I should remove the local copy of Akregator first:

You rebuild akregator but it seems that it uses akregator_config_userfeedback.so from your system not you install directory. If you rebuild remove your akregator from your system first.

I run sudo apt-get remove akregator and sure enough, it removes userfeedback:

0190 Reading package lists... Done
0191 Building dependency tree... Done
0192 Reading state information... Done
0193 The following packages were automatically installed and are no longer required:
0194   kuserfeedback-doc libkf5messageviewer-plugins libkuserfeedbackcore1
0195   libkuserfeedbackwidgets1 qml-module-org-kde-userfeedback
0196 Use 'sudo apt autoremove' to remove them.
0197 The following packages will be REMOVED
0198   akregator
0199 0 to upgrade, 0 to newly install, 1 to remove and 25 not to upgrade.
0200 After this operation, 6,495 kB disk space will be freed.

After running cmake ., I could see I need to install kuserfeedback-dev with:

0201 sudo apt-get install kuserfeedback-dev

After this I rebuild Akregator to ensure it's built against local sources:

0202 make clean
0203 make
0204 sudo make install # Install metakit

Now I am re-running the RAM study to see if the issue replicates. Lastly, I update the ticket and we see how things go.

Debugging (2021-08-09)

After running Akregator for about 24 hours, we finally see we are able to replicate the issue when building from source (and in debug mode):

24H RAM usage study

24H RAM usage study

Next we want to run Akreagtor under valgrind so that we can get more useful debug information:

0205 valgrind --leak-check=full -v --track-origins=yes --log-file=akregator-vg-2021-08-09.txt ./bin/akregator

Again, running with --track-origins=yes is insanely slow, to the point where I'm not sure the program is usbale and able to replicate the original issue.

Lastly, we update the ticket.

The next steps will be:

Debugging (2021-08-10)

The ticket has been updated with the valgrind output.

The next few days will likely be busy, hence I will need to move onto other projects. I will give the person responding on the ticket a chance to check things over and make recommendations - as I'm essentially just stabbing in the dark at this point.

Debugging (2021-08-13)

No further updates at this time - except I may have a branch of Akregator to checkout with a different archive back-end that should reduce RAM usage. Apparently, the default behaviour of Akregator is to try and store everything in RAM in order to keep some speed in the GUI.

Given how fast disks are (even old IDE drives), they should be more than fast enough to fetch some feed titles in a fast (undetectable) time. A design for any program that means it slowly uses more and more RAM over time - seems awful and broken. I'm beginning to believe that I might be better off rolling my own feed agreagtor. I don't have the time for that right now, but I really suspect it really is the right way forwards.

Debugging (2021-09-02)

Again, no further updates to report. Still not heard anything else from the developer. As time goes on, the idea of rolling my own feed agregator becomes more attractive. I suspect that the software would require a large overhaul as this is a design flaw - currently it tries to store everything in RAM.

I'm going to have to neglect this from my mind as I have more important things to work on. If I come back to it randomly in the future and there is still no update, I will consider it a lost cause.

Conclusion

Stay tuned for more!