I/O patterns on ELF binary initialization

[ I meant to finish this article much earlier, but i had to leave it in a half-written draft state for quite some time due to other activities ]

One particular issue that arises with big binary files is that of I/O patterns. It looks like it is not something under the usual scope for both ld.so and programs/libraries, but it can have a dramatic influence on startup times. My analysis is far from complete, would need to be improved by actually investigating ld.so code further, and so far has been limited to libxul.so built from mozilla-central, with the help of icegrind, on x86-64 linux.

As I wrote recently, icegrind allows to track memory accesses within mmap()ed memory ranges. As ld.so does mmap() the program and library binaries, tracking the memory accesses within these mmap()s allows to get a better idea at I/O patterns when an ELF binary is initialized. I only focused on local accesses within a single given binary file (libxul.so) because the case of Firefox loading many files at startup is already being scrutinized, and because most of the non-Firefox files being read at Firefox startup (system and GNOME libraries) are most likely already in the page cache. I also focused on that because it was an interesting case that could be helpful to understand how big (and maybe less big) binaries may be affected by the toolchain (compiler, linker and dynamic linker) and possibly by some coding practices.

For the record, what I did to get these results is to use icegrind and elflog, with further additions to the "sections" file with some output from "objdump -h": I added sections that elflog wouldn't give (such as .plt, .hash, .dynsym, etc.), and even down to the entry level for the .rela.dyn section. The latter is particularly interesting because icegrind only outputs the first access for any given section. To see sequential accesses within that section, you need to split it in smaller pieces, which I did for .rela.dyn entries (each one being 24 bytes on x86-64). Uncommenting some parts of the icegrind code was also useful to track where some accesses were made from, code-wise.

Now, the interesting data:

One of the first accesses is a nullification of a few variables within the .bss section. The .bss section is usually an anonymous piece of memory, that is, a range of mmap()ed memory that is not backed by a file, and filled with zeroes by the kernel (I think it even does that lazily). It is used for e.g. variables that are initialized with zeroes in the code, and obviously, any code accessing these variables would be addressing at some offset of the .bss section. It means the section needs to start in memory at the offset it has been assigned by the linker at build time. This is actually where problems begin.

When the .bss section offset as assigned by the linker doesn't align on a page (usually 4KB), the mmap()ed .bss can't be at that location, and really starts on the next page. The remainder is still mmap()ed from the binary file, and ld.so will itself fill that part. As the .bss section doesn't start on a page boundary, any write at this location will trigger the kernel reading the entire page. This means one of the first set of data being read from the file is the end of the section preceding .bss, and the beginning of the following one. Most likely, respectively the .data and .comment sections.

While this probably doesn't matter much when the binary file is small, when it is big enough, reading in a non sequential manner will trigger hard disk seeks, and we all know how they can hurt performance. Although thankfully, cheap SSDs should be coming some day, in the meanwhile, we still need to cope with the bad performance. The interesting part is, the .bss section is really empty in the binary file, so its "virtual" memory address could be anywhere. Why the linker wouldn't align it at a page boundary without having to resort to a linker script is beyond me.

The next accesses go back and forth between .dynamic, .gnu.hash, .dynstr, .gnu.version_r, .gnu.version and .dynsym. These are probably all related to symbol version resolution and DT_NEEDED library loading. While most of these sections are at the beginning of the file (not necessarily in the order they are read from), the .dynamic section is much nearer to the end, but way before .data, so that it won't even have been loaded as a by-product of the .bss section loading.

After that, the .rela.dyn section is read, and for each relocation entry it contains, the relocations are being applied. Relocations is one of the mechanisms by which position independent code (PIC) is made possible. When a library is loaded in memory, it is not necessarily loaded at the same address every time. The code and data contained in the library thus need to cope with that constraint. Fortunately, while the base address where the library is mmap()ed in memory is not necessary constant, the offsets of the various sections are still as codified in the binary. The library code can thus directly access data at the right offset if it knows the base offset of the library (which is what is done on the x86 ABI), or if it knows where the current instruction is located (which is what is done on the x86-64 ABI).

"Static" data (initialized in e.g. a const or a global variable, or, in C++ case, vtables), on the other hand, may contain pointers to other locations. This is where relocation enters the scene. The .rela.dyn section contains a set of rules describing where in the binary some pointers need to be adjusted depending on the base library offset (or some other information), and how they should be updated. ld.so thus reads all .rela.dyn entries, and applies each relocation, which means that while .rela.dyn is being read sequentially, reads and writes are also performed at various places of the binary, depending on the content of the .rela.dyn entries.

This is where this gets ugly for Firefox: there are near 200000 such relocations. On x86-64 an entry is 24 bytes (12 on x86), and each of these is going to read/write a pointer (8 bytes on x86-64, 4 on x86) at some random (though mostly ordered) location. The whole .rela.dyn section not being read ahead, what actually happens is that it is read in small batches, with seeks and reads of other data at various locations in between. In libxul.so case, this spreads over .ctors, .data.rel.ro, .got, and .data. The relocation entries are somehow ordered by address to be rewritten, though they occasionally jump backwards. Some of these relocations also appear to be touching to .gnu.version, .dynsym and .dynstr, because their type involves a symbol resolution.

Once .rela.dyn relocation have been dealt with comes .rela.plt's turn. The principle is the same for this section: entries describe what kind of relocation must be done where, and how the result must be calculated. The scope of this section, though, is apparently limited to .got.plt. But before explaining these relocations, I'll explain what happens with the PLT.

The PLT (Procedure Linkage Table) is used when calling functions from external libraries. For example, in a Hello World example, the PLT would be used for calls to the puts function. The function making the call would in fact call the corresponding PLT location. The PLT itself, on x86 and x86-64, at least, consists of 3 instructions (I'll skip the gory details, especially for x86, where the caller needs to also set some register before calling the PLT). The first instruction is the only one to be called most of the time: it reads the final destination in the .got.plt section, and jumps there. That final destination, obviously, is not fixed in the library file, since it needs to be resolved by its symbol. This is why the two subsequent instructions exist : originally, the destination "stored" in the .got.plt section points back to the second instruction ; the first instruction will effectively be a nop (no operation), and the following instructions will be executed. They will jump into code responsible for symbol resolution, update of the .got.plt entry for the next call, and call of the real function.

But pointing back to the second instruction is like the pointers in static data we saw above : it's not possible in position independent code. So, the .rela.plt relocations are actually filling the .got.plt section with these pointers back to the PLT. There are a tad more than 3000 such relocations.

All these relocations should be going away when prelinking the binaries, but from my several experimentations, it looks like prelinking only avoids relative relocations, and not the others, while it technically could skip all of them. prelink even properly applies all the relocations in the binary, but executing the binary rewrites the same information at startup for all but relative relocations. That could well be a bug in either ld.so not skipping enough relocations or prelink not marking enough relocations to be skipped. I haven't dug deep enough in the code to know how prelinking works exactly. Anyways, prelinking is not a perfect solution, as it also breaks ASLR. Sure, prelink can randomize library locations, but it relies on the user or a cron job doing so at arbitrary times, but that's far from satisfying.

An interesting thing to note, though, is that a good part of the relocations prelinking doesn't rid us of in libxul.so (more than 25000) are due to the __cxa_pure_virtual symbol, which is used for, well, pure virtual methods. In other words, virtual methods that don't have an implementation in a given class. The __cxa_pure_virtual function is set as method in the corresponding field(s) of the class VTABLE in the .data.rel.ro section. This function is provided by libstdc++, and as such, is dynamically linked. But this function is just a dummy function, doing nothing. Defining an empty __cxa_pure_virtual function to be included in libxul.so makes these relocations become relative, thus taken care of by prelinking.

After all relocations occur, the library initialization itself can begin, and the content of the .init section is executed. That section, indirectly, executes all functions stored in the .ctors section. This includes static initializers, which are unfortunately called backwards, as Taras pointed out already. Each of these static initializers are also accessing various locations in the .data or .bss sections, which may or may not have already been loaded during the relocation phase. The execution of these initializers will also (obviously) read various pieces of the .text section (despite its name, it contains executable sections, i.e. functions code).

The initialization of the library ends there, and no access should happen until a function from the library is called. In libxul.so case, XRE_main is first called, then many other functions, but that's another story. All that needs to be remembered about the startup process past this point is that the .text section will be read heavily, as well as the various data, got, plt and dynamic symbol sections, in a very scattered way. While most of these sections may have been retrieved in memory already, as a byproduct of the various initialization processes described above, some may have not, increasing even more the need to seek at all places in the binary file.

Now the main problem with all these I/O patterns at startup, is that it seems the only way reorganizing the binary layout may have a visible impact is by considering all the above, and not only a part of it, because only addressing a part of it is very likely to only move part of the problem to a different layer.

All in all, making sure the relevant sections of libxul.so are read by the kernel before ld.so enters the game is a good short-term solution to avoid many seeks at startup.

2010-07-14 18:37:13+0900

mozilla, p.m.o

You can leave a response, or trackback from your own site.

41 Responses to “I/O patterns on ELF binary initialization”

  1. Boris Says:

    > there are near 200000 such relocations.

    Do we know what the sources are? Is it worth trying to reduce this number?

  2. Kurt Roeckx Says:

    The best way to speed up the start up time is to avoid relocations. I suggest you read http://people.redhat.com/drepper/dsohowto.pdf

  3. James Vega Says:

    The people.redhat.com link doesn’t work anymore. Use http://www.akkadia.org/drepper/dsohowto.pdf

  4. Taras’ Blog » My Startup Summary Says:

    […] most of icegrind so in addition optimizing startup, it can also facilitate investigative work like Mike Hommey is […]

  5. glandium Says:

    Boris: I think it’s worthwhile to try to reduce the number of relocations. On the other hand, 166700 out of 204807 are due to vtables… considering XPCOM relies on them, it may be hard, though the de-COM-tamination process may help a bit.
    Reducing the other relocations and grouping the vtables together may help reducing seeks, too.

  6. Digital Blog Says:

    And Is it worth trying to reduce this number?

  7. Jakub Jelinek Says:

    If you have __cxa_pure_virtual prelink conflicts, the only possible reason for that is that you are defining __cxa_pure_virtual somewhere earlier in the search scope than in libstdc++.so.6. That is not the usual case and you should generally avoid that. E.g. linking static libstdc++ into the binary would result in something like that. Just never do something like that. Looking at firefox in Fedora 14, it has just
    Relocation section ‘.gnu.conflict’ at offset 0x748 contains 173 entries:
    (nothing in the order of thousands).

  8. Mark Heltor Says:

    Many fake crypto trading platforms have successfully managed to con victims of their hard-earned money. A few weeks ago, I was contacted by a crypto trader on Instagram who claimed to be a manager on a trading platform, he convinced me to invest my money into crypto mining hoping to make huge profits from it, I was convinced when I saw my account balance on their platform, but when I wanted to withdraw my earnings, they kept requesting for money for tax and other unnecessary fees. I realized then that I was being catfished, I stopped all communication and went to the internet to do my research about crypto when I came across Firmwall Cyber Security Service, a well reliable crypto recovery company, I was very pleased with the reviews I saw about this crypto recovery company and I contacted them to assist me to recover my crypto, I gave them all the information and within a few hours, Firmwall Cyber Security was able to recover my crypto assets. I’m truly grateful for their service and I highly recommend them

    E-mail = Firmwallcyber@techie. com
    Web = https://firmwallcyber.wixsite.com/firmwall
    Whats App = + 1 937 542 0667

  9. Rosemary Harper Says:

    IF YOU HAVE BEEN SCAMMED OF BITCOIN OR ANY OTHER CRYPTOCURRENCY, GET ASSISTANCE FROM THE WIZARD WILLIAMS RECOVERY TEAM.

    Scams involving bitcoin are getting increasingly sophisticated, and a lot of people fall for them. It can be rather frustrating, but if you’ve fallen for a bitcoin or USDT scam, don’t feel guilty. Instead, report the incident, , and seek the assistance you require. get your money back from fraudsters or Bitcoin, USDT, or other cryptocurrencies. The legitimate procedure to recover your stolen or defrauded money is to visit Wizard Williams Recovery Team if you have ever been the victim of a scam and would like to get your money back. I once lost roughly $210,000 in early January to a fraudulent broker who assured me of a 100% monthly return on my investment. A few days later, I realised that it was a dubious investment, and I started to worry that my money was lost. Fortunately, a close friend introduced me to Wizard Williams Recovery Team, and this team was able to fully recover my stolen funds in a matter of hours.I hired them, and this has been the best online service I’ve ever had. The agency is worthy of the great reviews they have received here. For comparable assistance, get in contact right away at:

    Email: Wizardwilliams@mail.com
    WhatsApp:  +4,9,1,7,6,1,2,4,5,2,0,6,6.

  10. Antonio Wenner Says:

    LEGIT AND FAST WAY TO RECOVER YOUR BITCOIN/CRYPTO 2024

    I was scammed over ( $345,000 ) by someone I met online on a fake investment project. I started searching for help legally to recover my money and I came across a lot of Testimonies about ETHICREFINANCE Recovery Expects. I contacted them providing the necessary information’s and it took the experts about 27hours to locate and help recover my stolen funds. I am so relieved and the best part was, the scammer was located and arrested by local authorities in his region. I hope this help as many out there who are victims and have lost to these fake online investment scammers. I strongly recommend their professional services for assistance with swift and efficient recovery. They can reached through the link below.
    Email Address: ethicsrefinance @gmail com
    Telegram: @ethicsrefinance

  11. Myra Jay Says:

    Hi everyone, I am indebted to this psychic and I still can’t believe my husband is back to me even in more stronger bond than we have ever been, with the help of this psychic, his services are outstanding and wonderful, within 24hours of connecting with him I got a surprise call from my husband and that’s how a problem that almost destroyed my marriage was solved. I highly recommend Priest Ray to anyone that wants accurate spell casting to get his or her ex partner back. You can dm him via WhatsApp +12133525735 or email psychicspellshrine@gmail.com
    Kindly visit https://psychicspellshrine.wixsite.com/my-site

  12. Lyne Azabal Says:

    I Need A Hacker To Recover My Lost Investment? Go To Lost Recovery Masters

    Hello guys, I’m Lyne Azabal from France. In my investment with Acstons-Cc, I also lost 17 lakhs. Trading using the Acstons. cc app generated a profit of 197,153 USD after paying tax and a 25% fee, however they would not release my profit and all of my Bitcoins were frozen. I called my pals, who suggested I try LOST RECOVERY MASTERS, and to my astonishment they were able to assist me get back 89% of my money. Despite the fact that I didn’t get all of it back, I’m still really appreciative of LOST RECOVERY MASTERS
    For advice on recovering cryptocurrencies, get in touch with Lost Recovery Masters You can also ask them questions about any firm you wish to deal with.
    Contact them:  Whatsapp (+44(7537)-105921) Email (Support@lostrecoverymasters.com) Learn More https://lostrecoverymasters.wixsite.com/recoverbitcoin2024

  13. base dase Says:

    Real OR fake Novelty Passports, Drivers Licenses, ID cards , Visas, Diplomas and many other documentsOur team has years of experience producing real passports,passports and other identity documents. We use high quality equipment and materials to produce counterfeit documents. All secret features of real passports are carefully duplicated for our falsified documents.we are unique producer of quality documents.We offer only original high-quality fake passports, driver|s licenses, ID cards, stamps, Visa, school Diplomas and other products for a number of countries like:USA, Australia, Belgium, Brazil, Canada, Italian, Finland, France, Germany, Israel, Mexico, Netherlands, South Africa, Spain, United Kingdom. This list is not full.To get further more detailed information about our high quality Authentic and passports/driving licenses/id cards/ stamps, Visa, school Diplomas and other product
    Email ( lifiben@gmail.com )
    TEXT/CALL/WHATSAPP: 951-254-2452
    https://clean-documents.com

  14. Joyce Marie Says:

    HOW TO RECOVER YOUR CRYPTO / BITCOIN FROM SCAMMERS.
    I want to share to the whole world how GEO COORDINATES HACKER helped me recover my stolen money. I thought I was going to lose everything I had invested into this crypto platform that turned out to be a scam, I was devastated when I lost all my funds to a scammer. When it comes to recovering lost cryptocurrency, GEO COORDINATES HACKER is the greatest. They have the most skilled bitcoin recovery specialists to handle your request and address any issues you may be having with bitcoin recovery. You can rely on GEO COORDINATES HACKER to do all in its power to help you recover your stolen money. If you need any help contact them by using Via Email; (geovcoordinateshacker@proton.me)
    Email; (geovcoordinateshacker@gmail.com)
    Website; https://geovcoordinateshac.wixsite.com/geo-coordinates-hack

  15. Andy Edward Says:

    HOW CAN I RECOVER LOST FUNDS FROM TRUST WALLET ? CONSULT CAPTAIN WEBGENESIS.

    Crypto Recovery Services – Through Captain WebGenesis / Legitimate Cryptocurrency Recovery Expert.

    Another Successful Bitcoin Recovery, My Spouse and I are another successful client of Captain WebGenesis among many. Numerous good reviews have been written about how the Crypto Recovery Expert has been utilizing his recovery skills to help people regain their happiness by getting their assets and Bitcoin back from dishonest and scammy cryptocurrency trading organizations. We were severely scammed by some scammers online, so I had to get in touch with them after reading about many of the fantastic services they offer. Fortunately, everything worked out well and Captain WebGenesis was able to get back the money we believed the Fraudsters had taken from us. Reach Captain WebGenesis via
    WhatsApp at +1 701) 314-27-29 or email at (Captainwebgenesis@hackermail.com)

    Website; http://Www.captainwebgenesis.com

  16. Jonathan Bill Says:

    HIRE A CERTIFIED CRYPTO RECOVERY EXPERT \ CAPTAIN JACK CRYPTO RECOVERY
    In the world of cryptocurrency, the security of Bitcoin (BTC) transactions is of utmost importance. However, despite the best precautions, there may be instances where BTC is lost, misplaced, or stolen. This is where CAPTAIN JACK CRYPTO RECOVERY comes into play, providing effective tools and solutions for BTC recovery.

    Bitcoin (BTC), the pioneering cryptocurrency, has taken the financial world by storm. Its meteoric rise, coupled with tales of overnight riches, has lured investors of all stripes. However, before diving headfirst into the alluring realm of BTC, a sober understanding of its inherent risks and potential pitfalls is crucial. Bitcoin (BTC) has emerged as one of the most popular and widely discussed investment options in recent years, attracting a growing number of individuals seeking financial opportunities in the digital realm. However, venturing into the world of BTC investments comes with its own set of risks and pitfalls that require careful consideration.

    This is no typical group of tech-savvy people, this is CAPTAIN JACK CRYPTO RECOVERY. They are the digital equivalent of superheroes, stepping in to save the day if your priceless Bitcoin (BTC) is misplaced or taken. CAPTAIN JACK CRYPTO RECOVERY is committed to helping people and companies recover their Bitcoin and reclaim their peace of mind. They do this by using their advanced technology and skills. One day, you become thrilled to check on your Bitcoin investment when you get up, only to discover that it has vanished. Disappeared. vanished out of sight. Not merely because of its monetary value, losing Bitcoin may be extremely distressing. Bitcoin is a symbol of a new era in digital commerce that gives users power. BTC recovery is essential because of this. Regaining control and safeguarding the future of digital currencies are more important than just the money. Modern BTC recovery strategies are based on cutting-edge technology, in contrast to ancient recovery methods that would entail consulting a wise old magician or shaking a crystal ball.
    CAPTAIN JACK CRYPTO RECOVERY uses a variety of strategies, including working with law enforcement, blockchain forensics, and advanced data analysis. These methods make sure that, in the pursuit of Bitcoin recovery, no digital stone is overlooked. CAPTAIN JACK CRYPTO RECOVERY is your best bet for getting your lost Bitcoin back. How can someone reach out to CAPTAIN JACK CRYPTO RECOVERY? use the information you see below.

    Email info – captainjackcrypto@europe.com

  17. Sarah Says:

    Nothing has seemed more satisfactory to me in recent times than the unmatched and professional services received from SWIFT RECOVERY FIRM when I contacted them following a friend’s recommendation regarding the $340,000 I had lost through a wrong Cryptocurrency investment with a wrong crypto platform. I thought nothing much could really be done before I wrote to them but still decided to give it a try just so I can be sure. Though way beyond my expectations, I got marvelled at the outcome as SWIFT ROCOVERY was able to track and recover my lost money. They are the best recovery firm to handle your case as I can testify to their topnotch services and highly recommend them to anyone going through scam to be sure of a swift retrieval of their lost investment funds, it was a very difficult time for me and my family, I was depressed and even gave up hope of getting my money back until SWIFT RECOVERY intervened in my case turning around my story to a triumphant ending. Contact details ( swiftrecoveryservice006 @gmail dot com or Whatsapp :  +1 (786) 684-0501

  18. Marie Faucet Says:

    Can those who have fallen victim to fraud get their money back? Yes, you might be able to get back what was taken from you if you fell prey to a fraud from an unregulated investing platform or any other scam, but only if you report it to the relevant authorities. With the right plan and supporting documentation, you can get back what you’ve lost. Most likely, the individuals in control of these unregulated platforms would attempt to convince you that what happened to your money was a sad accident when, in fact, it was a highly skilled heist. You should be aware that there are resources out there to help you if you or someone you know has experienced one of these circumstances. During my online lookup,i found Refundd Polici Recovery Services and they assisted me in the best way and recovered my stolen crypto assets worth hundreds of thousand dollars which was mainly my entire life savings i had planned out for my retirement with the documents i provided. They are reputable firm and strictly means business in helping make recoveries of stolen digital assets/scams and reducing the menace caused by these unregulated cyber thieves.
    You can contact them on whatapp +1 (605) 963-9055 and email on Refunddpolici AT gmail.com.
    Do not let the perpetrators of this hoaxes get away with ruining you mentally and financially.

  19. Roosevelt Lester Says:

    Lost crypto investment to scammers can be recovered if handled by professionals, and that’s where Cyber Genie Hack Pro comes in. Having to deal with this recovery folks made me realize the endless possibilities that the internet yields if used in the right way. These folks successfully recovered every penny lost to this false crypto investor, forever indebted to these peeps. Find them on Telegram: CYBERGENIEHACKPRO

  20. Violet Mason Says:

    NEVER GIVE UP, HAVE FAITH IN WIZARD WILLIAMS TO RESTORE ALL OF YOUR LOST CRYPTOCURRENCY.

    Have you lost all hope of ever getting your money back from scammers? I am delighted to inform you that you can absolutely get your money recovered. On the other hand, WIZARD WILLIAMS RECOVERY and his team’s knowledge is required for this. Before I came across the incredible recovery hacker WIZARD WILLIAMS RECOVERY, who helped me retrieve my stolen bitcoins, I had given up on ever making money from cryptocurrency trading after losing about $187.000 to a fake broker. For assistance with cryptocurrency recovery, get in touch with WIZARD WILLIAMS RECOVERY.

    EMAIL. wizardwilliams@mail.com

    WHATSAPP: +4,9,1,7,6,1,2,4,5,2,0,6,6.

  21. Melena Hill Says:

    Hello everyone, with so much joy in my heart today I wanna give thanks to this lottery spell psychic psychicspellshrine.wixsite.com/my-site, years have gone by without a fruitful reward over playing the lottery daily, I have spent a lot of tips on how to win the lottery, but when I came across Priest Ray my life changed for good. I requested a lottery spell, and he cast the spell and sent me the lucky winning lottery number which I played with and now I am a proud lottery winner with the help of Priest Ray. I recently won $3,000,000 and I am making this known to everyone of you who have been trying all day to win the lottery that there is an easy way to win, believe me this is the new way to win the lottery with better price amount, if you have ever been searching for away to win lottery email: psychicspellshrine@gmail.com or WhatsApp: +12133525735

  22. Emily Faye Says:

    I have been suffering from Herpes for the past 1 years and 8 months, and ever since then I have been taking series of treatment but there was no improvement until I came across testimonies of Dr. Silver on how he has been curing different people from different diseases all over the world, then I contacted him as well. After our conversation he sent me the medicine which I took according .to his instructions. When I was done taking the herbal medicine I went for a medical checkup and to my greatest surprise I was cured from Herpes. My heart is so filled with joy. If you are suffering from Herpes or any other disease you can contact Dr. Silver today on this Email address: drsilverhealingtemple@gmail.com or Whatsapp +2348120513902

    .////

  23. james belich Says:

    Winning the lottery was part of my dreams, I tried so hard to win big but all to no avail, until I came across Dr Lucas online who made my dreams come through and made me win 10 million dollars. I was a logistics manager who lives in Lancaster, S.C. and works about an hour’s drive away, in Charlotte, N.C., I stopped at a store to buy a scratch-off lottery ticket during my lunch break, because Dr Lucas gave me all the assurance that the numbers are not going to fail after I did all he asked me to do. Dr lucas is a powerful Dr that is on a mission to eradicate poverty from people’s lives and i have confirmed that by winning $10 million with the numbers he provided for me, it is my promise to tell the world about my experience with Dr Lucas and that’s what I’m doing now, you can win the lottery fast with the help of Dr Lucas he is tested and trusted Email: Drlucasspelltemple@gmail.com or WhatsApp +234 904 794 3567 he will help you.

  24. james belich Says:

    Winning the lottery was part of my dreams, I tried so hard to win big but all to no avail, until I came across Dr Lucas online who made my dreams come through and made me win 10 million dollars. I was a logistics manager who lives in Lancaster, S.C. and works about an hour’s drive away, in Charlotte, N.C., I stopped at a store to buy a scratch-off lottery ticket during my lunch break, because Dr Lucas gave me all the assurance that the numbers are not going to fail after I did all he asked me to do. Dr lucas is a powerful Dr that is on a mission to eradicate poverty from people’s lives and i have confirmed that by winning $10 million with the numbers he provided for me, it is my promise to tell the world about my experience with Dr Lucas and that’s what I’m doing now, you can win the lottery fast with the help of Dr Lucas he is tested and trusted Email: Drlucasspelltemple@gmail. com or WhatsApp +234 904 794 3567 he will help you

  25. Jin Xue Says:

    I came accross this post at the perfect time while in my thankful mood, I really want to appreciate Edith and the incredible team at REFUND POLICI RECOVERY SERVICE Wassap : + 1 ( 6 5 7 ) 2 6 2 4 4 8 2″ for saving my life and giving me another chance to make the right decisions. Navigating the aftermath of falling victim to a romance scam that threatened my financial security was one of the most challenging experiences of my life. The skammer was on the verge of draining my entire savings, leaving me feeling vulnerable and distraught. Not until my coworker randomly found out the level of mess i got myself into, and referred one of her old friend “Edith” to me as she was into tech , Edith demonstrated remarkable professionalism, empathy, and determination. Her assurance was a beacon of hope in an otherwise dark situation. I was surprised at their efforts and the team’s unmatched skills, I was able to recover every single penny i lost to the skammer. This outcome was something I barely dared to hope for at the start of this journey. Beyond the financial recovery, the support I received was instrumental in helping me regain my confidence and peace of mind.REFUND POLICI RECOVERY SERVICE’s dedication to their clients and their passion for justice in the crypto space is truly commendable. It’s clear that their team are not just in the business of recovery; they’re in the business of restoring hope and dignity to those who have been wronged.To anyone finding themselves in a similar situation, feeling lost and uncertain where to turn, I will highly recommend REFUND POLICI RECOVERY SERVICE. They even offer many other tech services like hacking of phone and monitoring a partner and other similar jobs. Thank you, Edith, for everything. Your work not only recovers lost assets but also helps mend spirits shattered by deceit. I am eternally grateful. One good turn deserves another, so i will leave their contakt below just incase anybody need such good services.
    Emeil: r e f u n d p o l i c i (AT) g m a i l (DOT) c o m
    Telgram: REFUNDPOLICI
    Thanks Again.

  26. Matthew Gordon Says:

    Winning the lottery is what I’ve always dreamt of, I tried so hard to win big but no avail, until a friend of mine told me about Dr Benjamin who had helped her strike a jackpot, when I first contacted him I was so skeptical about it but after he assured me that he will help me win, my hope came alive. The follow day he undergone me through some processes which I did with utmost trust and diligence, he gave me some lottery numbers and instructed me to purchase a lottery ticket, later that same day I purchased $50 ticket for the 500X The Cash?scratch-off game and hit on one of its 160 prizes of $1 million. The Lottery said he chose to?receive his?winnings as a one-time, lump-sum payment of $820,000 and bought the ticket from ?Riverwalk BP at 1304 Prudential Drive and I won the sum of $1,000,000. All thanks to Dr Benjamin for making my dream come through, he can help too contact him today at drbenjaminlottospell711@gmail.com or whatsapp him +17066036031 You can visit his website for more info https://dr-benjamin.com

  27. paul Says:

    HOW CAN I GET A LEGIT HACKER TO RECOVER MY LOST BTC/ETH/USDT/ OPTIMISTIC HACKER GAIUS IS THE BEST AND RELIABLE.

    My name is PAUL ANDERSON, here’s my recommendation…………….
    I have been in search of a good hacker for over 2 years, Recently I was looking through some websites; then I got OPTIMISTIC HACKER GAIUS, he was highly recommended on the website so i decided to try him out, he was able to carry out all the task i gave him within 24 hours, He also helps with: Binary Recovery, Files Recovery, Crypto investment, Password, Bypass / Recovery Malware Removal / Criminal Record Expunge, Blank ATM Card, Social Media Hack, Remote Mobile Monitoring & Hacking, Credit Repair and Private Key Reset. he is very good and reliable you can reach out to him if you are in need of the service of a good hacker

    Contact:
    Optimistichackergaius@seznam.cz

    Whatsapp:+,.1..6..0…1..4..6..0..9..;;4..7..7*

    Visit: https://optimistichackegaius.com

  28. Henry Says:

    RECOVERING LOST, HACKED, OR STOLEN BITCOIN /OPTIMISTIC HACKER GAIUS My Bitcoin wallet was hacked by fraudulent blockchain impersonators on Instagram who claimed to be from Blockchain. I was deceived into believing they were from Blockchain and gave them access to my wallet, stealing a total of 7.6859 BTC. It was all I had left in finances, so I was stunned and on the verge of unconsciousness. After that, I did my research online and discovered a post review about OPTIMISTIC HACKER GAIUS specialist. I explained my loss in a direct message to the specialist. So, in just two days, he assisted me in getting my bitcoin back. His knowledge enabled me to identify the offenders and assisted me in launching the rehabilitation scheme. and because of his knowledge, the perpetrators were also identified. I hope that I was also able to assist someone. Get in contact with OPTIMISTIC HACKER GAIUS Specialist if you want to get your money back from any online scam.

    Contact details
    email… optimistichackergaius @ seznam.cz

    call or use WhatsApp at +1..6..0…1..4..6..0..9…4..7..7.

    Visit: https://optimistichackegaius.com

  29. Phillip Wright Says:

    I understand that losing access to your wallet or bitcoins can be stressful, or stolen usdt crypto from your wallet address. Still, I must emphasize the importance of exercising caution and seeking assistance through legitimate channels. I have seen and perused cryptocurrency scams. It reminds me of some heartache experience when I lost 547k dollars to a fake online crypto investment scam when I invested a huge amount of money. I searched online and came across a recovery expert whom I contacted via { http://www.linuxcybersecurity.com } who in a short while helped recover my scammed money, and I was pleased.
    I will advise you to contact the Website above for assistance,{LCS} he is fast and reliable.

  30. Andrew Kriver Says:

    Choosing A Reliable Bitcoin Recovery Service: iBolt Cyber Hacker Verified Agency

    iBolt Cyber Hacker is known for their expertise in recovering lost or inaccessible Bitcoin funds. With a team of skilled professionals and state-of-the-art technology, they have successfully helped numerous clients recover their digital assets securely and efficiently. When choosing a Bitcoin recovery service, it is essential to consider factors such as reputation, experience, security measures, and success rate. iBolt Cyber Hacker excels in all these areas, making them a top choice for individuals seeking reliable assistance with Bitcoin recovery. Whether you have lost your Bitcoin wallet password, experienced a hacking incident, or encountered any other issue preventing access to your funds, iBolt Cyber Hacker can provide the expert help you need. Trustworthy, efficient, and dedicated to client satisfaction, they are a dependable partner in navigating the complexities of Bitcoin recovery.

    More Info:

    Email: Support@iboltcyberhack.com

    Contact/Whatsapp: +39 350 929 0318

    Website: https://iboltcyberhack.com/

  31. frank Says:

    ADMIN HACKER GAIUS: IS THE BEST COMPANY FOR CRYPTO CURRENCY RECOVERY AND HOW TO RECOVER LOST CRYPTO FUNDS

    The investment scheme initially drew me in with the promise of large profits and the guarantee of a safe investment. But as time passed, I came to see that the investment was a complete fraud. I was left feeling dissatisfied and powerless after losing access to my money. It was a terrible experience, and I thought I was out of choices. I looked up a few cryptocurrency recovery businesses online and read customer endorsements and reviews after doing some investigation. To assist me in getting my investment back, I had to discover a reliable provider. I gave it some thought before deciding to get in touch with ADMIN HACKER GAIUS who received great feedback and endorsements. Their professionalism and promptness really pleased me, and they offered me hope by thoroughly explaining the recuperation process. After he recovered well, I agreed to review the fantastic work he did. admin Hacker Gaius is a great resource for any fraud victim looking for assistance in recovering their lost cryptocurrency.

    Contact details:
    E-mail: admin@weicherthallmarks.com

    WhatsApp: +1 401 219-5530

  32. Kelvin Jackson Says:

    Choosing a Trustworthy Bitcoin Recovery Service: Paradox Recovery Wizard Verified Agency

    Paradox Recovery Wizard is well-known for their ability in recovering Bitcoin monies that have been lost or are inaccessible. With a team of qualified personnel and cutting-edge technology, they have successfully assisted several clients in recovering their digital assets in a secure and quick manner. When selecting a Bitcoin recovery service, examine reputation, experience, security measures, and success rate. Paradox Recovery Wizard succeeds in all of these aspects, making it an excellent alternative for anybody looking for dependable help with Bitcoin recovery. Whether you’ve forgotten your Bitcoin wallet password, been hacked, or faced another issue that prevents you from accessing your cash, Paradox Recovery Wizard can provide the skilled assistance you require. Trustworthy, efficient, and more They are a reliable companion in negotiating the complications of Bitcoin recovery.

    More Information:

    Email address: paradox_recovery@cyberservices.com.

    Contact/Whatsapp number: 39 351 222 3051.

  33. AMISON GIBSON Says:

    Hello everyone, Ghost Champion Cyber Recover Services, are the only true and guaranteed cryptocurrency recovery professionals. I want to tell you about how I almost had my life ruined by a man who conned me out of $809,278.10 after I fell for a romantic hoax. I fell for this con man, and it damaged my connection with my family. My life savings were stolen by this scammer I met on a dating website while he seemed to be in trouble, and I assisted him numerous times since I was continually sending money through Bitcoin and bank transfers. As a result, I had to declare bankruptcy. I borrowed money to help him, but I later learned that he was a con artist. After a buddy of mine showed me an article about Ghost Champion Cyber Recover Services and how they are capable of recovering any crypto coins Bitcoin, USDT, ETH, Dogecoin, and bank transfer funds. from any type of con artist, i reach out to Ghost Champion Cyber Recover Services discuss my situation, and send all the information of the transfer payment I made to the scam. In less than 24 hours I was able to get through everything and to my great surprise, they quickly retrieved most of what I had lost to my Bitcoin wallet address. If you find yourself in such difficulty, get in touch with them right away. to recover your scammed funds back to your bank account or to your bitcoin wallet address Email: ghostchampionwizard@gmail.com all thanks to Ghost Champion Cyber Recover Services Contact Asap on Telegram on : https://t.me/WizardGhosthacker , Email : ghostchampionrecovery@cyberservices.com

    Cyber security Advertising agency #CryptoSafety #ScamAwareness #BlockchainSecurity

  34. Amelia Hensworth Says:

    Spyhost Cyber Security Company , in fact, never disappoint. Few weeks ago, I did invest $320,000 in bitcoin with a phony company. A few days after I made the investment, I was shocked to discover on my dashboard that it had climbed to almost $420,000. Never knew my happiness was about to diminish as I attempted to withdraw the money and was turned down. I repeatedly emailed help, but they never got back to me. I decided not to let that get away from me and started thinking of methods to get back what was properly mine. On social media, I came across folks who had also been duped by a similar investment fraud and had successfully retrieved their money via Spyhost Cyber Security Company . The experts at Spyhost Cyber Security Company are excellent and their top notch hacking prowess was proven of which I am now a living testimony because all my invested funds were recovered within 48 hours. If you want to recover money/ lost digital assets from con artists, consider using Spyhost Cyber Security Company. For additional information about their services, reach out to them through the contact means below;
    Email; Spyhost@cyberdude .com

    Whatsapp: (571) 478-7636

  35. FRANK WARD Says:

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

  36. FRANK WARD Says:

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

    YES YOU CAN RECOVER ALL YOUR MISSING CRYPTOS. CONTACT THE HACK ANGEL NOW

    It is best to set aside time to discover a legitimate professional who can assist you in recovering your stolen or lost cryptocurrency, such as bitcoins, rather than falling victim to other amateur hackers who cannot complete the task. THE HACK ANGELS is the most trustworthy and reputable blockchain tech specialist you can deal with to reclaim what you’ve lost to scammers. They helped me get back on my feet, and I’m grateful. Contact them right away to retrieve your misplaced coins.

    Email. hackangel@cyberdude.com

    WHATsAP: +1 203,309,3359

    Web: https://thehackangels.com

    Thank you THE HACK ANGEL

  37. david Says:

    HOW TO RECOVER STOLEN CRYPTO

    Would you like to get your lost Bitcoin back?
    Do you want the money that was taken from you in Bitcoin returned?
    You need to get in contact with the appropriate agency in order to accomplish this.
    The HACKER GAIUS Group is an expert in cases pertaining to fraudulent investment schemes and cryptocurrency. It is a private detective agency that also recovers funds.
    To get started, report your concern to Optimistic Hacker Gaius using the contact information provided below.

    WhatsApp at +1..6..0..1..4..6..0..9..4..7..7.

    email… optimistichackergaius (@) seznam.cz

    website: https://optimistichackegaius.com

  38. kevin Says:

    A Guide to Recruiting a Hacker to Retrieve Stolen Bitcoin.

    My gratitude is insufficient for the outstanding assistance rendered by the renowned OPTIMISTIC HACKER GAIUS. I lost a significant sum of my hard-earned money—$258k worth of Bitcoin—to an online romance fraud, which crushed me. But this group of OPTIMISTIC HACKER GAIUS professionals quickly helped me out and proved their proficiency with crypto recovery. They recovered my lost money in an unexpectedly quick amount of time. Their unwavering efforts and dedication to ensuring client happiness really kept me afloat financially. I would heartily suggest them to anyone facing a comparable situation. Their exceptional assistance and capacity to retrieve significant quantities of cryptocurrency are unparalleled. Details about how to reach us are provided below.

    WhatsApp at +1..6..0..1..4..6..0..9..4..7..7.
    email… optimistichackergaius @ seznam.cz

    website: https://optimistichackegaius.com

  39. Edward Gilbert Says:

    My Review: How to Recover Your Lost or Stolen Bitcoins – iBolt Cyber Hacker

    iBolt Cyber Hacker offers solutions to help regain control of digital assets, Whether your bitcoins were stolen through hacking, lost due to forgotten keys, or inaccessible for any other reason, iBolt Cyber Hacker has the experence to help you. Their team use advanced blockchain technology with sophisticated cybersecurity techniques to navigate the complexities of cryptocurrency recovery.

    I Truly Recommend Them

    More Info:
    Email: Support @ iboltcyberhack . com
    Contact/Whatsapp: +39 350 929 0318
    Website: https : // iboltcyberhack . com /

  40. Felix peter Says:

    Recovery Lost Cryptocurrency, USDT, BTC, Lost Wallet, and Optimistic Hacker Gaius

    I was defrauded of a substantial amount of money on an online trading platform. I had absolutely lost hope until I discovered an OPTIMISTIC HACKER GAIUS on Reddit, which allowed me to recover all of my lost money in addition to any alleged gains. Permit me to share my story and how I managed to receive my money back.

    WhatsApp, +1..6..0..1..4..6..0..9..4..7..7.

    email…. optimistichackergaius @ seznam.cz

  41. Fredric Says:

    NEED A HACKER WHO CAN RECOVER STOLEN OR SCAM*ED BTC?

    If you need genuine hackers who can recover stolen or scam*ed Cryptocurrency OPTIMISTIC HACKER GAIUS. got you covered, they’re secured Hackers from USA who deal with cyber issues of all kind, such as WhatsApp hack, Facebook hack, Gmail hack, mobile Phone hack, Accounts hack, Erase Criminal Records, Websites Hack, Bitcoin Trade recovery, and other hack relating to cyber issues. Contact info:

    WhatsApp at +1..6..0..1..4..6..0..9..4..7..7.
    email… optimistichackergaius @ seznam.cz

    website: https://optimistichackegaius.com

Leave a Reply