[kde-linux] baloo_file_extractor

Discussion:

[kde-linux] baloo_file_extractor - huge disk usage?

Mark Knecht

2015-08-13 21:36:35 UTC

Hi,
I'm not sure if this is a Gentoo issue or a KDE issue so I'll start
on the KDE list. Thanks in advance.

I have a problem that's developed recently. The basic symptom is
the machine slows down horribly at some point in the afternoon with
the disk activity light is full on - no blinking at all. This results
in all applications - Virtualbox, Chrome, etc. - essentially ceasing
to operate correctly and KDE to sometimes locking up for 5-10 minutes.
Due to recent messages from Gentoo devs requiring baloo I've started
looking there.

When I'm in this state where disk activity appears to be 100% I see
large wait percentages in top. (3 CPUs in this state right now.)
Running iotop it appears that something called baloo_file_extractor is
spending 99.9% of its time waiting.

While writing this message the disk light has gone back to no
activity, the machine is responsive and the baloo_file_extractor app
is no longer using io.

So, the basic question is how do I turn this off or at least get it
under control as it's causing huge problems when it occurs?

If this isn't the right list please advise where is better. Maybe
it's a distro issue.

Thanks,
Mark

c2RAID6 ~ # top
top - 14:26:00 up 4:53, 3 users, load average: 4.66, 4.57, 4.43
Tasks: 225 total, 1 running, 224 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.3 us, 1.0 sy, 0.0 ni, 2.8 id, 92.7 wa, 0.0 hi, 3.1 si, 0.0 st
%Cpu1 : 1.0 us, 0.0 sy, 0.0 ni, 0.0 id, 99.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 0.0 id,100.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.7 us, 0.0 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 24685212 total, 16563312 used, 8121900 free, 454088 buffers
KiB Swap: 12582904 total, 0 used, 12582904 free. 13843164 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2574 root 20 0 213512 120496 86040 S 1.3 0.5 8:01.83 X
1111 root 20 0 0 0 0 D 1.0 0.0 2:21.59
md3_raid6
3 root 20 0 0 0 0 S 0.3 0.0 0:30.64
ksoftirqd/0
1092 root 0 -20 0 0 0 S 0.3 0.0 0:50.42
kworker/0:1H
3304 mark 20 0 3514860 193656 84836 S 0.3 0.8 4:38.51
plasma-desktop
3307 mark 20 0 4636 2036 1580 S 0.3 0.0 1:11.55 ksysguardd

c2RAID6 ~ # ps aux | grep baloo
mark 3303 0.1 0.1 335300 33272 ? SNl 09:34 0:18
/usr/bin/baloo_file
mark 12966 2.0 0.1 239424 39836 ? DN 14:15 0:01
/usr/bin/baloo_file_extractor 159184 159183 159182 159181 159180
159179 159178 159177 159106 159105 159104 159103 159102 159101 159100
159099 159098 159097 159096 159095 159094 159093 159092 159091 159090
159089 159088 159087 159086 159085 159084 159083 159082 159081 159080
159079 159078 159077 159076 159075
root 12978 0.0 0.0 8928 928 pts/1 S+ 14:16 0:00 grep
--colour=auto baloo
c2RAID6 ~ #

c2RAID6 ~ # iotop -ob -d 10
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
1111 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [md3_raid6]
Total DISK READ : 114.25 K/s | Total DISK WRITE : 5.37 M/s
Actual DISK READ: 114.25 K/s | Actual DISK WRITE: 5.40 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
13036 idle mark 0.00 B/s 5.34 M/s 0.00 % 99.99 %
baloo_file_extractor 159074 159073 159072 159071 159070 159069 159068
159067 159066 159065 159064 159063 159062 159061 159060 159059 159058
159057 159056 159055 159054 159053 159052 159051 159050 159049 159048
159047 159046 159045 159044 159017 159016 158990 158802 158721 158720
158719 158718 158717
1111 be/4 root 0.00 B/s 0.00 B/s 0.00 % 72.28 % [md3_raid6]
1124 be/3 root 0.00 B/s 2039.91 B/s 0.00 % 8.45 % [jbd2/md3-8]
12459 be/4 mark 407.98 B/s 7.17 K/s 0.00 % 0.25 % plugins
[Chrome_CacheThr]
8407 be/4 mark 113.85 K/s 27.19 K/s 0.00 % 0.03 % VirtualBox
--comment TradeStation Stable --startvm
49bef29d-6408-470e-a0c3-f3930f98e502 --no-startvm-errormsgbox
[AioMgr0-N]
7707 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.02 % [kworker/1:0]
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More info: http://www.

Duncan

2015-08-14 00:02:06 UTC

Permalink

Post by Mark Knecht
I'm not sure if this is a Gentoo issue or a KDE issue so I'll start
on the KDE list. Thanks in advance.

Arguably, it's a kde as upstream issue, but gentoo offers the possibility
of mitigating it, arguably more than most binary distros. =:^)

Post by Mark Knecht
I have a problem that's developed recently. The basic symptom is
the machine slows down horribly at some point in the afternoon with the
disk activity light is full on - no blinking at all. This results in all
applications - Virtualbox, Chrome, etc. - essentially ceasing to operate
correctly and KDE to sometimes locking up for 5-10 minutes.
Due to recent messages from Gentoo devs requiring baloo I've started
looking there.
When I'm in this state where disk activity appears to be 100% I see
large wait percentages in top. (3 CPUs in this state right now.) Running
iotop it appears that something called baloo_file_extractor is spending
99.9% of its time waiting.

Yeah... that behavior is why I decided the (small to me) benefits of
semantic-desktop in general weren't worth the down side. Tho the
tradeoff is arguably different for each person/use-case, the basic
problem remains the exact same one it was back before the turn of the
century[1], when the first thing most computer-knowledgeable folks did to
an MS Office installation was disable its indexer functionality.

Post by Mark Knecht
While writing this message the disk light has gone back to no
activity, the machine is responsive and the baloo_file_extractor app is
no longer using io.
So, the basic question is how do I turn this off or at least get it
under control as it's causing huge problems when it occurs?
If this isn't the right list please advise where is better. Maybe
it's a distro issue.

While it might be possible to throttle baloo to some extent (I'm not
actually sure because like the nepomuk before it, it's banished from my
system), and at least back in the nepomuk era when I rid myself of it, it
was in theory possible to turn off at runtime, at least in my experience,
actually building kde with USE=-semantic-desktop (obviously not an option
on distros where the binaries come pre-built, thus my remark above about
gentoo making a workaround easier) and without related flags and packages
[2], then depcleaning them from the system, made kde *SURPRISINGLY*
faster. "Surprisingly", because I /thought/ the runtime turnoff actually
did so, and the build-time disable and depclean was primarily to clean up
the dependencies. But /something/ was obviously still sucking
performance as I'm not kidding, after getting it off the system entirely,
it was as if I suddenly had a couple extra cores or had upgraded at least
a half a GHz in cpu speed, which REALLY surprised me. It felt like I
guess the MS users must feel after they get the malware cleaned off!

While I can't guarantee similar effects to everyone, ridding your system
of semantic-desktop and the baloo indexer and akonadi, should at minimum
guarantee that it won't run and effectively lock you out of normal
operations on your system for minutes at a time.

And actually, the gentoo/kde devs have recently cleaned up the old
akonadi deps in the old pre-akonadi-kmail kdepim-4.4.x series as well, so
you shouldn't even require akonadi and via it, semantic-desktop, for the
non-akonadi kmail/kdepim stuff, either. The other alternative being to
switch to non-kdepim solutions for mail and related (IM, feed-reader,
etc). I've been rather happy with claws-mail since I switched to it
here, altho the migration from kmail isn't as simple and easy as I would
have liked.

If you have detail questions and would prefer to take it to the gentoo-
desktop list (or gentoo-amd64, but the desktop list is more topical),
that's fine, or continue here if you like, since it /is/ kde-linux
related, and I'm on both this the kde-general and kde-linux lists and
those gentoo lists.

---
[1] MS at the turn of the century: I haven't allowed MS on my systems
since the introduction of eXPrivacy shortly after the turn of the
century, as it crossed a line I simply wasn't going to cross. Funny that
many of the people who objected to eXPrivacy back then, like frogs in a
heated pot, are now swearing by eXPrivacy, as MS Windows 10 goes even /
further/ down that path. I guess MS is counting on them to protest a bit
as they did last time, but ultimately, continuing to sit in the pot until
they're well boiled. <shrug> Well, I for one, didn't! For that reason
I've little knowledge beyond what I read of what MS does today, nearly a
decade and a half later.

[2] Semantic-desktop related flags/packages: Back then, rasqual,
virtuoso, nepomuk, akonadi, etc, some of which will no longer be valid
with baloo, but I'm not sure if the deps have actually gone away or have
only been replaced with other deps. I did have to keep strigi as some kde
packages (kdelibs and kde's systemsettings are the two I have installed
that still dep on it) apparently link to its headers without a disable
option, but with all its backends turned off, it couldn't actually do
anything, so other than having to still have it installed to link to,
it's pretty impotent.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/faq

Mark Knecht

2015-08-14 00:24:21 UTC

Permalink

Hi Duncan

Post by Duncan

Post by Mark Knecht
I'm not sure if this is a Gentoo issue or a KDE issue so I'll start
on the KDE list. Thanks in advance.

Arguably, it's a kde as upstream issue, but gentoo offers the possibility
of mitigating it, arguably more than most binary distros. =:^)

:-)

Post by Duncan

OK, so the KDE documentation suggests that it can be turned
off using System settings -> Desktop search and at the bottom there's
an entry for 'Enable Desktop Search' so I've unchecked that and will
see how it goes tomorrow.

<SNIP>

Post by Duncan
While it might be possible to throttle baloo to some extent (I'm not
actually sure because like the nepomuk before it, it's banished from my
system), and at least back in the nepomuk era when I rid myself of it, it
was in theory possible to turn off at runtime, at least in my experience,
actually building kde with USE=-semantic-desktop (obviously not an option
on distros where the binaries come pre-built, thus my remark above about
gentoo making a workaround easier) and without related flags and packages
[2], then depcleaning them from the system, made kde *SURPRISINGLY*
faster. "Surprisingly", because I /thought/ the runtime turnoff actually
did so, and the build-time disable and depclean was primarily to clean up
the dependencies. But /something/ was obviously still sucking
performance as I'm not kidding, after getting it off the system entirely,
it was as if I suddenly had a couple extra cores or had upgraded at least
a half a GHz in cpu speed, which REALLY surprised me. It felt like I
guess the MS users must feel after they get the malware cleaned off!

In that regard there are two packages that use the flag, dolphin & gwenview.
Depending on how things go tomorrow just having disabled it I'll give rebuilding
those two a try.

Post by Duncan
While I can't guarantee similar effects to everyone, ridding your system
of semantic-desktop and the baloo indexer and akonadi, should at minimum
guarantee that it won't run and effectively lock you out of normal
operations on your system for minutes at a time.

One would hope!

Post by Duncan
And actually, the gentoo/kde devs have recently cleaned up the old
akonadi deps in the old pre-akonadi-kmail kdepim-4.4.x series as well, so
you shouldn't even require akonadi and via it, semantic-desktop, for the
non-akonadi kmail/kdepim stuff, either. The other alternative being to
switch to non-kdepim solutions for mail and related (IM, feed-reader,
etc). I've been rather happy with claws-mail since I switched to it
here, altho the migration from kmail isn't as simple and easy as I would
have liked.
If you have detail questions and would prefer to take it to the gentoo-
desktop list (or gentoo-amd64, but the desktop list is more topical),
that's fine, or continue here if you like, since it /is/ kde-linux
related, and I'm on both this the kde-general and kde-linux lists and
those gentoo lists.

I'll try these ideas out over the next few days and see what happens. I
generally just use locate to find things so none of this overhead is necessary
for my normal day to day life.

Thanks!

Cheers,
Mark
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
Mo

Mark Knecht

2015-08-14 00:28:03 UTC

Permalink

Post by Mark Knecht

Post by Duncan
If you have detail questions and would prefer to take it to the gentoo-
desktop list (or gentoo-amd64, but the desktop list is more topical),
that's fine, or continue here if you like, since it /is/ kde-linux
related, and I'm on both this the kde-general and kde-linux lists and
those gentoo lists.

I'll try these ideas out over the next few days and see what happens. I
generally just use locate to find things so none of this overhead is necessary
for my normal day to day life.

BTW - I found this on the KDE site:

[QUOTE]
Baloo/Semantic Search is eating 100% CPU! What do I do?

Just wait. Certain files are very hard or even impossible to Index. At
the moment, this includes for example text files of over 50 megabyte.
When Search finds these, it will try for a fixed time. When it fails,
it will try to find out what file is broken and disable indexing it in
the future. As it indexes files in batches of about 40, it has to find
the problematic file by indexing that bunch in parts: first
half/second half, index problematic half in pieces again, until the
file is found. This can take up to 30 minutes of heavy cpu usage.
Unfortunately, while Baloo will not start to index a new batch of 40
files while on battery power, it continues to determine the broken
file while on battery. This behaviour has been fixed in in KDE
Applications 4.13.1 (it will stop indexing immediately when the power
cord is unplugged) and the time the search for each file can take has
been reduced to about 10 minutes. The Semantic Search team is working
on improving the indexing tools to handle more difficult files.
[/QUOTE]

Amazing...JUST WAIT!

- Mark
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More i

Duncan

2015-08-14 02:59:01 UTC

Permalink

Post by Mark Knecht
In that regard there are two packages that use the flag, dolphin &
gwenview. Depending on how things go tomorrow just having disabled it
I'll give rebuilding those two a try.

I use both apps, and as I said, have USE=-semantic-desktop, so I can tell
you what the effect is. In dolphin, there's an extra tab in the
properties dialog with semantic-desktop, that's not there (it used to be
there but empty, now it seems it's not there at all) without it. Since
it's not there I don't remember what it was named, but it had additional
"meta" information about the file, for instance on images, the exif, etc
information included in the file, like what camera was used and its
settings, what image-editing software was used, etc. Similarly for media
files, it would contain info on codecs used, id3 tag info like artist/
album/etc on mp3s, etc.

In gwenview it's the same general info, presented IIRC (again, I've had
semantic-desktop off long enough the details of what it was like with it
on are getting blurry) on the meta-information side-panel tab.

Back in kde3, this sort of information was made available without
semantic-desktop, which kde3 didn't have (IIRC it got the info using
filetype specific kioslaves back then), but with kde4, they migrated all
that functionality to semantic-desktop, meaning you have a choice between
indexing it all and accepting the negatives for doing so whether you want
to or not, or not having that information exposed at all; there's no
choice to have it available when curious, but not actually indexed.

So now if you still want to have the information available and/or
separately indexed, you can use alternative (often gtk-based, sometimes
separate non-kde qt-based) apps not tied to kde and its semantic-desktop.

In kde-frameworks/5 (which I've tried but which was still kwin-crash-
looping last I did so; I need to try again...) they've supposedly made
things a lot more modular, so I'm hopeful it'll continue to be possible
to build dolphin, gwenview, etc, without semantic-desktop support, tho
I'm not sure. What would be /real/ nice would be if the reading and
display of this info was modularly separate from the indexing, in kde5,
making the information display available again, as it was in kde3,
without having to deal with the indexing cruft, but I'll be satisfied if
it simply doesn't require it. If it requires it, then... it'll be a lot
easier for me to leave kde behind entirely with the kde4 -> kde5
transition than it was with the botched kde3 -> kde4 transition, as it
and stuff like the akonadified kmail, etc, have me off of the nearly
complete kde standardization I had in the kde3 era, and I'm using
primarily just the kde4 plasma core desktop, these days. Plasma/kwin/
superkaramba would indeed be a hassle to replace and suitably configure
the replacement, but the plasma upgrade would be about the same for that
one, and replacing the other two should it be necessary shouldn't be
anything like as difficult as what I've already replaced during the life
of kde4.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More info: http:/

Vojtěch Zeisek

2015-08-14 06:59:12 UTC

Permalink

Post by Mark Knecht
Hi,
I'm not sure if this is a Gentoo issue or a KDE issue so I'll start
on the KDE list. Thanks in advance.
I have a problem that's developed recently. The basic symptom is
the machine slows down horribly at some point in the afternoon with
the disk activity light is full on - no blinking at all. This results
in all applications - Virtualbox, Chrome, etc. - essentially ceasing
to operate correctly and KDE to sometimes locking up for 5-10 minutes.
Due to recent messages from Gentoo devs requiring baloo I've started
looking there.
When I'm in this state where disk activity appears to be 100% I see
large wait percentages in top. (3 CPUs in this state right now.)
Running iotop it appears that something called baloo_file_extractor is
spending 99.9% of its time waiting.

There has been several bugs reported on that topic...
https://bugs.kde.org/show_bug.cgi?id=349096
https://bugs.kde.org/show_bug.cgi?id=233471
https://bugs.kde.org/show_bug.cgi?id=333655
https://bugs.kde.org/show_bug.cgi?id=332317
https://bugs.kde.org/show_bug.cgi?id=333774
VojtÄch

--
VojtÄch Zeisek

Komunita openSUSE GNU/Linuxu
Community of the openSUSE GNU/Linux

http://www.opensuse.org/
http://trapa.cz/

Mark Knecht

2015-08-14 17:24:02 UTC

Permalink

Hi Vojtěch.

On Thu, Aug 13, 2015 at 11:59 PM, Vojtěch Zeisek

Post by VojtÄch Zeisek

Wow, you've been dealing with this for a long time! (#333655) Sorry to
hear that.

Thanks for providing the links.

I Was wondering last night if some of my problem is due to the way
I've set up shared directories in Virtualbox. A number of directories
inside of Gentoo are seen as remote drives inside of my Windows VMs
allowing the VMs to share data with each other. A lot of my Baloo
problems seem to come in the afternoon, after the stock market closes
and the VMs are saving lots of stock price data to these directories.
I end up with 1000's of files rewritten every afternoon.

Anyway, I appreciate the response and am going down the path suggested
by Duncan, along with simply disabling indexing, assuming that option
work. We'll see over the next few days.

Cheers,
Mark
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More

Mark Knecht

2015-08-15 19:49:57 UTC

Permalink

Post by Mark Knecht
Hi Vojtěch.
On Thu, Aug 13, 2015 at 11:59 PM, Vojtěch Zeisek

Post by VojtÄch Zeisek

Wow, you've been dealing with this for a long time! (#333655) Sorry to
hear that.
Thanks for providing the links.

So the 1 day test of disabling indexing in KDE System Settings failed,
although not nearly as badly as it had failed earlier. In this
incantation the machine essentially locked up for about 15-20 minutes
but eventually came back.

I've now rebuilt KDE with -symantec-desktop and done an emerge
--depclean to clean out the old stuff. The file indexing option is
gone from System Settings and there's no baloo left on the system
other than old config files. I'll give this setup a few days and see
how it goes. Hopefully it will be ok but if not I'll likely try some
other environment like xfce to determine if the root cause of this is
in KDE, Linux or maybe one of the Windows VMs running in Virtualbox.

Thanks for the help & ideas.

Cheers,
Mark
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/f

Mark Knecht

2015-08-25 18:31:58 UTC

Permalink

On Sat, Aug 15, 2015 at 12:49 PM, Mark Knecht <***@gmail.com> wrote:
<SNIP>

Post by Mark Knecht
So the 1 day test of disabling indexing in KDE System Settings failed,
although not nearly as badly as it had failed earlier. In this
incantation the machine essentially locked up for about 15-20 minutes
but eventually came back.
I've now rebuilt KDE with -symantec-desktop and done an emerge
--depclean to clean out the old stuff. The file indexing option is
gone from System Settings and there's no baloo left on the system
other than old config files. I'll give this setup a few days and see
how it goes. Hopefully it will be ok but if not I'll likely try some
other environment like xfce to determine if the root cause of this is
in KDE, Linux or maybe one of the Windows VMs running in Virtualbox.
Thanks for the help & ideas.
Cheers,
Mark

Reporting back in case someone searches for this issue in the future.

With symantec-desktop turned off this problem has not shown up in a
week so as far as I can tell this was the root cause and everything is
now OK.

Thanks for the help.

Cheers,
Mark
___________________________________________________
This message is from the kde-linux mailing list.
Account management: https://mail.kde.org/mailman/listinfo/kde-linux.
Archives: http://list