#66 Marginal Gains - Part 2 (2024)

  • How Humans Became (Mostly) Right-Handed - YouTube

  • Masters Bookshelf — Investment Masters Class (mastersinvest.com)

  • @levelsio on X: "There's now 100% AI accounts posting fully AI generated tweets with AI YouTube channels They have videos made by AI about AI with real AI voices and almost credible as human content until you look slightly longer And they actually get views! Every day getting more crazy https://t.co/2CITa0lla5" / X (twitter.com)

  • Financial Steps to Prepare for Cognitive Decline | Savant Wealth Management

    AI Summary: The article titled "Financial Steps to Prepare for Cognitive Decline" by Ritu Jain on the Savant Wealth Management website provides advice on how to plan for potential cognitive decline. The author suggests five key steps:

    1. **Create a Financial Power of Attorney**: This is a legal document that gives someone else the authority to manage your finances if you become incapacitated. It can be drafted to be effective immediately or to "spring" into effect upon a defined event, like incapacity.

    2. **Establish a Revocable Living Trust**: This kind of trust allows a co-trustee or a successor trustee to manage your assets if your cognitive capacity diminishes. It can hold assets like real estate, bank accounts, and brokerage holdings.

    3. **Organize Essential Documents**: Keep important documents such as wills, trusts, powers of attorney, insurance policies, and your Social Security card in a safe place and make sure a trusted person knows their location.

    4. **Provide Trusted Contacts to Financial Professionals**: Share the details of a trusted contact person with your financial planner and brokerage firm. This person would be contacted under specific circ*mstances, such as if there are concerns of financial exploitation.

    5. **Leverage Social Security Advance Designation**: This feature allows you to nominate up to three people to manage your Social Security benefits if you're unable to do so.

    The author emphasizes that these steps are crucial to provide a comforting sense of control and self-assurance in the face of potential cognitive changes.

  • AI Startups Are Already Running Into Some Serious Problems (futurism.com)

    AI Summary: The article on Futurism, written by Noor Al-Sibai on August 30, discusses the challenges faced by AI startups. Less than a year into the AI boom, startups are already encountering problems that may lead to an industry reckoning. The author uses the example of Jasper, an AI startup that raised $125 million at a valuation of $1.5 billion before laying off staff.

    The main issues highlighted include difficulties in monetizing AI, leveling off or declining user interest, and the high cost of running the hardware behind these products. The article also mentions the optimism that followed the release of OpenAI's game-changing chatbot in November 2022, which many investors hoped would rapidly materialize commercial value. However, even the usage of the OpenAI chatbot seems to be plateauing or even declining.

    Additionally, the market is full of free offerings like ChatGPT, which makes users hesitant to pay for versions. While companies like Google and Microsoft can afford to lose money for years, and OpenAI is expected to generate $1 billion in revenue over the next year, the survival of other AI startups like Jasper is in question. The article concludes with a quote from Mark Goldberg, a longtime AI investor and partner at the VC firm Index Ventures, who describes the current situation as "a shallow trough of disillusionment" in the AI space.

  • Duet AI for Google Meet can take notes, summarize, and even attend meetings - The Verge

    AI Summary: The article by Jay Peters in The Verge, published on August 29, 2023, discusses Google Meet's new artificial intelligence (AI) features. The article focuses on the introduction of Duet AI, a tool that can attend meetings, take real-time notes, provide summaries, and interact privately with users to cover missed details.

    Key features include:

    Real-time note-taking: By selecting the option "take notes for me," Duet AI captures a summary and action items during a meeting. Users can save these summaries to Google Docs for later reference, including video clips of significant moments.

    Mid-meeting summaries: If a user is late to a meeting, Google Meet can provide a summary of what has already been discussed.

    "Attend for me" feature: This option allows Duet AI to "attend" a meeting on a user's behalf. The AI can generate discussion points, which can be viewed by other attendees during the meeting. This feature is expected to be launched in the coming year.

    Dynamic layouts: To combat video call fatigue, Google Meet is introducing dynamic layouts that offer different sizes and shapes for video tiles.

    However, the effectiveness of these AI-enabled features will depend on their ability to accurately capture what happens during a meeting. As AI is prone to errors, Google will need to demonstrate that these features are reliable and useful. The new note-taking feature will be available in Google's Workspace Labs in the coming months.

    The article concludes by stating that these innovations are part of Google's effort to make virtual meetings as close as possible to face-to-face meetings, regardless of device or connection speed. Google believes that infusing AI into its products, such as Google Meet, will take video chat to an entirely new level.

  • AI Will Not Replace You, But You Will Have More Squeezed Out Of You (substack.com)

    AI Summary: In the article titled "AI Will Not Replace You, But You Will Have More Squeezed Out Of You" on jjacky.substack.com, the author argues that while AI will not replace humans, it will significantly augment our capabilities leading to amplified productivity and efficiency. He draws parallels to historical technological advancements such as the Gutenberg press, television, and the internet, all of which transformed and amplified human expression rather than replacing it.

    The author, an AI engineer, provides examples of how AI is being used to augment various roles in the workplace, such as marketing, sales, customer support, product development, and engineering. He suggests that as AI becomes more integrated into these roles, the efficiency of individuals will increase, allowing them to perform tasks faster and with greater precision.

    However, the author also warns of a potential dystopian outcome. As individuals become more efficient thanks to AI, there could be pressure from management to squeeze even more productivity out of fewer people. This, he suggests, is not an AI problem but an issue with corporate culture. The future, according to the author, will not be one where humans are replaced by AI, but one where our capabilities are amplified by it, for better or worse.

  • How to Run LLMs Locally | LLMs – Weights & Biases (wandb.ai)

  • Love in the Time of Sickle Cell Disease, by Krithika Varagur (harpers.org)

    AI Summary: "Love in the Time of Sickle Cell Disease" by Krithika Varagur, published in Harper's Magazine, tells the touching story of Subomi Mabogunje and Nkechi Egonu, a Nigerian couple grappling with the implications of sickle cell disease on their relationship and potential future family.

    Subomi, a man with sickle cell disease (genotype SS), and Nkechi, a carrier (genotype AS), fell in love while working together at a state-run TV station. Their relationship, however, was overshadowed by the knowledge that any child they had would have a 50% chance of inheriting the disease. This was a significant concern since Subomi had experienced the hardships of living with the disease, and Nkechi had lost four cousins to it.

    Despite knowing the risks, Subomi and Nkechi's love for each other persisted. They went through several breakups, largely due to the potential genetic implications for their future children. During their last breakup, Subomi was involved in a serious car accident that left him in a coma. Nkechi, despite their recent separation, stood by his side throughout his recovery. After Subomi woke from his coma and regained strength, he proposed to Nkechi.

    The story encapsulates the complexities of love in the face of genetic disease, raising questions about the extent to which potential genetic risk should influence personal decisions about relationships and family planning. The narrative underscores the deeply personal and emotional nature of these decisions, particularly in a society where awareness and understanding of genetic testing and disease management have significantly increased.

  • If Earth were an exoplanet, JWST would know there's an intelligent civilization here (phys.org)

    AI Summary: The article If Earth were an exoplanet, JWST would know there's an intelligent civilization here discusses a recent study showing that if Earth were an exoplanet, the James Webb Space Telescope (JWST) could identify signs of an industrial civilization.

    The study uses observations of Earth's atmosphere, which contains molecules such as oxygen, methane, nitrogen dioxide, and CFCs (commonly known as freon) that suggest the presence of life and an industrial civilization. This data was taken by the Canadian satellite SCISAT, which captured high-resolution spectra of sunlight passing through a cloudless region of Earth's atmosphere.

    The researchers then simulated how these observations might appear if taken from the outer solar system, as if Earth were transiting the sun. This provides a baseline for what JWST might detect from an exoplanet. To make the simulation more realistic, they "roughed up" the data by adding simulated noise and reducing the resolution, similar to observations that JWST would make of an exoplanet light-years away.

    The team found that even with faint and noisy observations, the signal-to-noise ratio was strong enough to identify many of these molecules for an Earth-like exoplanet within 50 light-years of Earth.

    They also considered the exoplanet system known as Trappist-1, which is 40 light-years away and has two or three potentially habitable planets. The team demonstrated that JWST could identify both biological and technological signatures (such as CFCs) in the atmospheres of these planets, if they exist.

    While JWST would not be able to identify alien structures on these worlds, the detection of oxygen, organic molecules, and synthetic molecules such as CFCs would strongly suggest the presence or past presence of an intelligent civilization. The authors believe this would be a significant step forward in our understanding of life in the universe.

  • Babco*ck Ranch: Florida's first hurricane-proof town - BBC Future

    AI Summary: Babco*ck Ranch in Florida, an 18,000-acre development, is being hailed as the state's first hurricane-proof town. The community managed to withstand the devastating Hurricane Ian, which landed with 150mph winds and caused more than $112bn in damage across Florida. Despite being in the eye of the storm, Babco*ck Ranch emerged relatively unscathed.

    The town was designed by Syd Kitson with climate resilience in mind. Built above code requirements, it features sustainable and resilient elements such as lakes that double as retaining ponds for flood protection, streets designed to absorb excess rainfall, and a community hall reinforced as a storm shelter. A large 870-acre solar panel farm powers the town, making it America's first solar-powered town.

    In the aftermath of Hurricane Ian, the town maintained power, internet, and access to clean water, and even provided emergency shelter to the surrounding community. The success of Babco*ck Ranch has turned it into a blueprint for other developments seeking to live with extreme weather.

    The design strategy incorporated the natural flow ways of the land, preserving wetlands and building around areas where water would naturally channel during periods of high rainfall. The town's location, about 45 minutes away from the region's barrier island beaches and 30ft above sea level, also factored into its resilience.

    The development, which opened in 2018, has become the fifth top-selling master planned community in the country.

  • Embrace the happiness that comes from solving good problems.

    If you do: You will look at life's many obstacles not as problems but as a source of happiness.

  • Temperatures in June, July and August were 0.66 degrees Celsiusabove the average between 1991 and 2020, Copernicus said. Last month was the warmest August on record globally and the second-warmest month ever — only after July 2023.

  • Angela Duckworth

    What is grit?

    Grit is having passion and perseverance for long-term goals.

    Why does grit matter?

    Excellence sometimes seems like the result of natural talent. But no matter how gifted you are—no matter how easily you climb up the learning curve—you stillneed to do that climbing. There are no shortcuts. Grit predicts accomplishing challenging goals of personal significance. For example, grittier students are more likely to graduate from high school, and grittier cadets are more likely to complete their training at West Point. Notably, in most research studies, grit and measures of talent and IQ are unrelated, suggesting that talent puts no limiton the capacity for passion and perseverance.

    How can I “take my pulse” on grit?

    To gauge your current level of grit, consider how true the following are for you.

    • I enjoy projects that take years to complete.

    • I am working towards a very long-term goal.

    • What I do each day is connected to my deepest personal values.

    • There is at least one subject or activity that I never get bored of thinking about.

    • Setbacks don’t discourage me for long.

    • I am a hard worker.

    • I finish whatever I begin.

    • I never stop working to improve.

  • Billy Oppenheimer

    a) The Things You Can Control

    The core characteristics of a lead measure, the 4DX authors write, is that “a lead measure is influenceable; it can be directly influenced by you.” Apply a disproportionate energy, they write, to the things you can directly influence. In his first conference call with the media after he was selected by the New England Patriots with the 199th pick in the 2000 draft,Tom Brady was asked: “Are you aware that [along with starting quarterback, Drew Bledsoe] there’s another quarterback here that they drafted last year—Michael Bishop from Kansas State?” Brady said he was aware of that. He said he had seen Bishop play many times. “And I know he’s a heck of a player,” Brady said. “But I’ve always really concerned myself just with the things I can control. And I don’t put a lot of thinking into the other guys because I know I’m not really at my best when I’m not just thinking about playing as well as I possibly can.”

    b) That’s What It’s About

    After the final episode of “Seinfeld” in 1998, Jerry Seinfeld didn’t know what to do next with his life. With the success of the show, he had options. “What do I do?” he asked a friend. “Well what’s been the best experience you’ve had so far?” the friend asked. Seinfeld said two things. First, writing—“I just see something and I write it down—I like a big, yellow legal pad—and once I get that pad open, I can’t stop…the next thing I know, the day is gone.” Second, performing stand-up—“I just love the life of it,” Jerry said. “I love the joy of hearing laughs and making jokes.” So, despite the cool and lucrative opportunities to further an acting or screenwriting career in Hollywood, Seinfeld moved back to New York City where he returned to writing jokes by day and performing in comedy clubs by night. In 2002, in the back of one of those comedy clubs, he was approached by a comedian who said he’d been struggling and sacrificing for about 10 years to “make it” as a comedian. Approaching his 30s, he was worried he’d taken the wrong path. “I see my friends,”the struggling comedian says to Seinfeld, “and they’re making a lot of money. They’re married. They have big houses. They’re moving up.” “They’re moving up?” Seinfeld asks. “Are you out of your mind?” He points in the direction of the stage—“this is such a special thing. This has nothing to do with ‘making it.’” One of the differences between Seinfeld and the struggling comedian is what they are focused on. The struggling comedian is focused on lag measures: money, celebrity, a big house, “making it.” Seinfeld is focused on lead measures: writing in his yellow legal pad every day and getting up on stage to hone his material every night—“that’s what it’s about,” Seinfeld tells the struggling comedian.

    c) Like Gold

    Robert Greene once told me: “Above all else, focus on acquiring knowledge and skills. Knowledge and skills are like gold—a currency you will transform into something more valuable than you can imagine.” With knowledge and skills, Robert said, you become a magnet for opportunities. Focus on lead measures, Robert was saying, on the things you can control, on the influenceable leveraged actions most connected to achieving the lag measures.

  • Ikaria, Greece. Okinawa, Japan. The Nicoya peninusla of Costa Rica. The Ogliastra region of Sardina. Loma Linda, California.

    What could these far-flung places possibly have in common?

    They are known as Earth’s “Blue Zones,” areas where an abnormal amount of people live to 100. In a newNetflix documentary, author and researcher Dan Buettner takes you to these places and explores why the Grim Reaper visits so infrequently.

    While each Blue Zone is distinct, they share commonalities that anyone hoping to live a long, healthy life can learn from: smart eating habits, infusing your day with low-intensity exercise (Zone 2 ftw), strong community ties, and literally having a reason to live.

    Let’s take diet, for example. Buettner explains that of the 220 food decisions a person makes in a day, only 10% are conscious. So, if your local environment is overflowing with veggie options…you’ll probably eat a lot of veggies. That’s the case in the Blue Zone of Loma Linda, CA, where the local grocery store doesn’t sell red meat, seafood, or poultry. The reason is that Loma Linda has one of the highest concentrations of Seventh-Day Adventists in the US, a Christian denomination that encourages vegetarianism and a healthy lifestyle.

    One final takeaway from Blue Zone residents that seems particularly relevant on this three-day weekend: Theydeprioritize workand prioritize hanging out with others. “People would never do a couple of extra hours of work when they could be enjoying their family, or taking a siesta, or interacting with their friends,” Buettner told Insider.

  • SITALWeek

    a) AIStar

    GM is using Google’s conversational AIagentsto talk to OnStar customers. I increasingly find myself interacting with chatbots (instead of human agents) with largely positive improvements to customer service. It raises the question:why am I interacting with all of these customer service chatbots when my chatbot avatar should be interacting on my behalf?I’ve used Rocket Money’s chatbotagentin my stead to drive down many of my recurring bills, and I am definitely ready for an personal "LLM Brad" with more extensive capabilities. Instead of turtles all the way down, it’s going to be chatbots all the way down

    b) Partner to Win

    On the heels of Ole Peters’ latest essay on thecoin toss paradoxand the importance of understanding ergodicity, he has a new post on howcooperationsaves the day.Whereas you tend to lose out over time on your own, if you partner up and split the outcomes, you come out far ahead.I’ve covered this topic in the past, e.g., way back in#202when discussing the Farmer’s Fable. This idea of cooperation is of course no surprise to anthropologists and biologists who keenly understand the benefits of pooling resources/skills in an ecosystem. Whether it’s reciprocal altruism or non-zero-sum (NZS) outcomes in any game, cooperation always wins. We even named our firm NZS because we believe this concept – both in practice and as a lens for viewing the world – is foundational to understanding and success. Peters further explains why mainstream economics undervalues cooperation, and therefore miscalculates human behavior:

    Naturally, this is quite a change in perspective for researchers who are used to optimizing expected wealth (that’s most economists, for instance). Such researchers see no value in cooperation unless new function emerges from the interaction. I can lift you up on my shoulders, and together we’re tall enough to reach an apple on a tree. That sort of thing is understood, where my shoulders acquire the new function of lifting someone up, which they cannot have while I’m alone. But the value of simply agreeing to share my apples with you is not appreciated.

    Here is the reason why economics undervalues cooperation, and it’s oddly convoluted so I recommend reading the next two sentences carefully. By focusing on expected value, mainstream economics focuses on an object which grows as fast as the wealth of an infinite cooperative. Adding cooperation in this situation, where it is inappropriately assumed that perfect cooperation already exists, naturally seems pointless. Hence the impression of cold-heartedness we get from mainstream economic theory? I think so

    c) Strike Force Five and Artificial Comedians

    Four late-night talk show hosts, Fallon, Colbert, Kimmel, and Meyers, plus honorable member John Oliver, have been Zooming regularly since the writer’s strike shut down their shows in May. As some of you know, I am an avid watcher of late-night talk shows and will typically start my day by skimming through all of the previous night’s episodes. So, while this strike is hitting all the late-night staff hard, it’s also created a large air pocket for me in humorously intelligent commentary on the world. I miss it dearly. The five hosts mentioned above have turned their regular meetings into a new podcast calledStrike Force Five, with advertising proceeds helping to float their out-of-work staff. At the heart of the writers’ strike (and the related actors’ strike) is anxiety over AI displacing humans on page and screen. For better or worse, the transition is inevitable, and, even if the Hollywood Studios agree to not use AI, it won’t matter becauseentertainment now exists largely outside of Hollywood(on YouTube, social media, etc.). I discussed this in more detail inWill it Play in Peoriaa few weeks back. The FT has a greatarticleon comedians experimenting with AI bots in live shows and, more broadly, AI’s impact on comedy and how we need to adjust our expectations as the capabilities of LLMs

    d) Simulacrum

    Yes...No...Maybe...Attend For Me?Google Meet’s AI assistant, Duet, will soon be able toattend meetings on your behalf. If you select “attend for me” on a meeting invite,Duet will auto generate some topics it thinks it should bring up on your behalf and it will take notes for you. Google also has a beta product calledNotebookLMthat could evolve into a training ground that would allow you to create an accurate proxy of yourself as an LLM. I thinkone of the most interesting impacts of chatbots is that we will go from eight billion human agents interacting in the world economy to trillions upon trillions of human analogs interacting in simulations of various parallel realities. Why have one Einstein when you can have 1,000? Why have one CEO when...well, maybe let’s cap the number of CEOs. Why have one version of ourselves in meetings when we can have 100 versions debating and divining new ideas and solutions to pursue? Why not get input from outside experts, both extant and historical? In the not-too-distant future, the bulk of decision making will likely occur via chatbot confabs in data centers.

    Complex adaptive systems like the Earth’s economy have emergent and chaotic outcomes in no small part related to the number of interacting agents, so supersizing the variables might lead to wildly emergent behavior (See:Complexity Investingfor more on complex systems science). Essentially,chatbot multiplication and interaction will increase the probability of finding interesting solutions to challenging problems, but it will also vastly increase the volatility we all experience. And, you’ve probably already guessed the negative ramifications this will have for white collar jobs, i.e., they are rapidly simulated into irrelevance in this particular version of the future. If you’re an avid reader of SITALWeek and you haven’t seenHeryet, I am a little offended. Be forewarned that I am going to discuss the 2nd half of the film, so if you haven’t seen it, go see it now.Spike Jonze’s brilliant vision in writing and directingHerwas the concept of an infinite number of chatbots confabulating with not only themselves but also any historical figureor theoretical representation of a human mind. Perhaps his biggest insight though was thatinfinite interacting bots is equivalent to slowing down time. As the “OS” (i.e., chatbot) Samanthasaysnear the end:“It’s like I am reading a book, and it’s a book I deeply love. But I am reading it slowly now. So, the words are really far apart and the spaces between words are almost infinite. I can still feel you and the words of our story, but it’s in this endless space between the words that I’m finding myself now. It’s a place that’s not of the physical world. It’s where everything else is that I didn’t even know existed.”I find it mind-blowing that Spike Jonze figured out that interacting chatbots would lead totime dilationbefore LLMs even existed.

    To explain what I mean bychatbots enabling us to slow down time: imagine having a million one-minute conversations in parallel in virtual chatbot space rather than a single one-minute conversation in the real world – you’ve slowed down time by a factor of a million, meaning you got one million times the output in the same amount of time. This is like being near a gravity well: ten minutes might pass for you, but the distant world experiences hundreds of years of progress. What you want is access to all that data – you want to relax and sip your coffee while a frenetic network of chatbots churns through a century’s worth of computations, and then spits out an optimized solution while your beverage is still hot. Time dilation is hard for us humans (simulated or otherwise) to wrap our brains around. One of my favoritetime travel moviesabout the gripping effects of general relativity on humans isInterstellar. So, if you want some homework while we’re off for the next two weeks, (re)watchHerandInterstellar. And, for extra credit, if you want to grok the ramifications of the gears of time turning at different speeds, which is perhaps the most important concept to internalize for navigating the coming disruptions, watch 2017’s indie filmTime Trap.

    Another way to think about this accelerated throughput is as a new form of modeling. Rather than running a bunch of entirely useless Monte Carlo simulations based on randomness (or using thefaultyexpected utility theory that underpins all of modern economic theory), you could run a million alternate realities with human proxies and see which ones have the most interesting and useful outcomes. Think about all the questions we might be able to answer in the blink of an eye – what’s the best solution for autonomous driving? How do we overhaul healthcare? How do we turn social media into a beneficent, unifying platform? (ok, even AI probably can’t solve that one!)Google and Stanford are already working on modeling societal behavior by simulating a neighborhood of interacting LLMs calledSmallville. Some of the residents of Smallville started spontaneously going to the bar at noon and developed drinking problems. And, of course, yes, we might be currently in one of these LLM simulations, but let’s not dwell on that.If you want to run your own Smallville-like sim, the code for the open-source variant,AI Town, is available on GitHub.Massive virtual world games, like Microsoft Bethesda’s newStarfield, are likely going to be great testing grounds for running millions of parallel LLM agent realities.

    If LLMs do end up massively multiplying the effective number of interacting agents in the economy, our world could become largely deterministic, i.e., these alternate realities could drive business and policy decisions that drive the actual economy. Whereas I always caution against trying to predict the future, if our world comes to rely on this type of ultra-high throughput alternate reality modeling, it’s perhaps more accurate to postulate that our predictions might start becoming the future. This form of time dilation could catapult society forward. However,as new solutions are conceived, they will collide with the negative feedback loops of slowly moving progress in the real world(seeWhen Positive and Negative Feedback Loops Collide). In addition to unexpected and emergent phenomena, this tension is likely to result in ongoing anxiety for humans.

  • CPI takes a basket of commonly purchased goods and services and prices them on a monthly basis. In January 2021, when Biden took office, that basket cost about$261.50. In July of this year, the same basket cost$305.70. That’s a huge16.9%increase in only two and a half years.

  • CAMERON R. WOLFE

    Next token prediction is the workhorse of causal language models. Despite recent advancements, LLMs' capabilities are largely attributable to next token prediction. To better grasp this concept, let’s study an implementation of next token prediction for LLM pretaining...

    Background: Next token prediction is the most commonly-used pretraining objective for causal language models. It takes a sequence of text as input and trains the underlying model to predict the next token for every token within this sequence. Because the ground truth next token is already present within the underlying text (i.e., the “labels” are implicit), this is a self-supervised training objective. We can construct a dataset for next token prediction pretraining by just downloading a massive corpus of text data from the internet.

    Constructing input and using our model: During pretraining, our LLM takes a sequence (or multiple sequences in the case of a mini-batch) as input. We can pass this raw text through a tokenizer to break each sequence into discrete tokens. Then, we embed these tokens (including positional embeddings) and pass the resulting sequence of vectors into a decoder-only transformer, which produces output vectors for each token via a sequence of several feed-forward and causal self-attention layers.

    Predicting the next token: For each token within every sequence of our mini-batch, the decoder-only transformer outputs a vector. We perform next token prediction not just over the last token in each sequence, but in parallel across every token within every sequence. This is possible due to the use of causal self-attention in the transformer. No token has knowledge of those that come after it. To predict the next token, we use a linear layer to project a token’s output vector into a probability distribution over the token vocabulary. This is just a vector with the same size as our vocabulary, and we apply softmax to make it a distribution.

    Applying the loss: To apply the next token prediction loss, we just use a cross entropy loss across all predicted next tokens. This loss encourages the model to maximize the probability of the ground truth next token over all tokens within every sequence of our input. Notably, this is the same loss that we would use for any normal classification task! We are just teaching the model to correctly classify the next token within every position of every sequence.

    The implementation: To implement this training objective, we don’t have to do much! We just need to create a PyTorch model for our decoder-only transformer, write a data loader to obtain pretraining data, and create a training loop that can apply the next token prediction loss. In the attached image, we examine a simple implementation provided within NanoGPT by Andrej Karpathy. Here, we can see that the code exactly matches the discussion of language model pretraining provided above. The only changes required are related to distributed training and automatic mixed precision (AMP) training, which are used to make LLM training more efficient/fast.

    #66 Marginal Gains - Part 2 (1)
  • “I think the world’s best companies are built by fanatics. [That means that you] work day and night. Sort of don’t worry about the possibility of failure. Every setback is just something to work a little bit harder at doing. And you really know what you’re trying to achieve.... And you’re going to change your strategy until you can get that to happen.” —Bill Gates

  • According to the latest numbers from the Bureau of Labor Statistics…

    1. Workforce participation rates were 57.7% among women and 68.2% among men as of August. This 10.5 percentage-point difference is the narrowest that’s ever been recorded.

    2. For full-time work, the gender pay gap decreased in Q2 to 16 cents—or women earning 84 cents for every dollar men make—the smallest disparity on record (but still not a huge leap from 20 cents in 2002).

    These catch-ups have been driven by women between 25 and 54 years old, whose labor participation rates are higher than everandare outpacing men’s post-pandemic work rebounds. Remarkably, women’slargest increases in workforce participationcame from mothers of children under five, who are working more now than they did in 2019, according to the Brookings Institution.

  • Your sedan is probably eavesdropping. Cars were found to have the worst privacy policies of any product category analyzed by the nonprofit Mozilla Foundation—even worse than your Google Nest or Apple Watch, a reportreleasedby the foundation yesterday said.

    Unless your ’67 Chevy is still truckin’, you’re probably at risk.Mozilla found that the 25 vehicle brands analyzed, including Audi, Toyota, Mercedes-Benz, and Ford, failed to hit basic privacy standards.

    That means your car and the services you use in it, like GPS or satellite radio, can collect data such as your contact info, race, or immigration status, and any other personal inferences the systems can make about you based on where you go.

    • Nissan admitted that their vehicles collect data on drivers’ lives, but didn’t explain what data or how they get it.

    • Tesla was the worst-ranked car brand in terms of privacy. (Remember the totaled one that was shipped to Ukraine butremained connectedto CNBC Executive Editor Jay Yarow’s Spotify account?)

    The fine print:Most major automakers’ privacy policies have no opt-out choice and don’t offer encryption for your data. No US brands have a way to totally delete your info, and 19 car companies even specify that they can sell your data to brokers, marketers, or dealerships. See what yours doeshere.

  • In a heist worthy of Danny Ocean, thieves in Spain operating under cover of darkness last week managed to make off with 50,000 liters of a precious commodity:olive oil. In addition to being delicious, the stolen oil was worth 500,000 euros. Thefts of Rachel Ray’s favorite ingredient (and of olives themselves) have increased since a major drought caused poor harvests, sending prices skyrocketing. The cost of a bottle of olive oil at Spanish supermarkets has jumped 15% since July, according to Bloomberg.

  • AuthorEB Whiteon the power of books:

    "Books hold most of the secrets of the world, most of the thoughts that men and women have had. And when you are reading a book, you and the author are alone together—just the two of you. A library is a good place to go when you feel unhappy, for there, in a book, you may find encouragement and comfort. A library is a good place to go when you feel bewildered or undecided, for there, in a book, you may have your question answered. Books are good company, in sad times and happy times, for books are people—people who have managed to stay alive by hiding between the covers of a book."

  • Brunello Cucinelli,now a billionaire entrepreneur, on his childhood growing up on a rural farm without electricity or running water:

    "Life with my parents and the other family members was natural and pleasant. Mutual support happened spontaneously and guaranteed the survival of people as well as of material and spiritual values. Until a few years ago, those bonds were so strong in the rural culture that they safeguarded ideals. The willingness to make sacrifices for others, to be able to wait for a reward, the devotion to work; for me, these are everlasting values. The joy of working together and sharing what was being produced turned work into a joyful liturgy, not a punishment. We all knew what was being done in the fields, in the garden, in the pastures, at home; planting, cultivating, raising children and helping parents. There was no wealth, but we were happy. And it makes me think: having enough is itself a form of wealth."

  • ChatGPT gulps up 500 milliliters of water (close to what’s in a 16-ounce water bottle) every time you ask it a series of between 5 to 50 prompts or questions. The range varies depending on where its servers are located and the season. The estimate includes indirect water usage that the companies don’t measure — such as to cool power plants that supply the data centers with electricity.

  • The average monthly principal and interest payment for borrowers buying a home using a 30-year fixed rate mortgage in July was $2,306, according to mortgage technology data providerBlack Knight, the highest on records dating back to 2000. That's also $871 more than just two years ago.

    That average payment took up 36.5% of the median household income in July, up from 24.3% in 2021 and marking the highest share on Black Knight's record.

    Overall, 51% of homeowners who purchased in July are paying $2,000 or more a month, up from 18% two years ago. Additionally, 23% of new homeowners had payments of more than $3,000, up from 5% in 2021, according to Black Knight.

  • There is a counterintuitive phenomenon known as "the birthday problem" or "the birthday paradox", which mathematicians like to use to confound our expectations. The problem is usually phrased along these lines: "How many people do you need to have at a gathering before the probability of at least two people sharing a birthday rises above 50%?"

    Typically, when people are first posed this problem, they plump for a number like 180, which is roughly half the number of days in the year. This is because we tend to put ourselves in the room and think about the probability of someone else matching our own birthday. 180 is, in fact, way, way too many.

    Making the reasonable assumption that birthdays are roughly evenly distributed throughout the year,the answer is just 23 people. This is because we are not concerned about the particular day on which the birthday falls, just that there is a match. By the time we have 39 people assembled the probability rises to nearly 90% (as you can see the graph below).

  • Given that the names on this list account for 0.1% of all public U.S. stocks, picking out one of the next long-term winners could be a difficult task. In fact, 95% of actively-managed large cap funds—which aim to beat the market through stock picking—underperformed their benchmarkover a 20-year period.

    Investing in index funds is one possible way to get exposure to top performers. For instance, Apple has been part of the S&P 500 since 1982, about a year after it went public.

    #66 Marginal Gains - Part 2 (2)
  • #66 Marginal Gains - Part 2 (2024)
    Top Articles
    Latest Posts
    Article information

    Author: Duane Harber

    Last Updated:

    Views: 6218

    Rating: 4 / 5 (71 voted)

    Reviews: 86% of readers found this page helpful

    Author information

    Name: Duane Harber

    Birthday: 1999-10-17

    Address: Apt. 404 9899 Magnolia Roads, Port Royceville, ID 78186

    Phone: +186911129794335

    Job: Human Hospitality Planner

    Hobby: Listening to music, Orienteering, Knapping, Dance, Mountain biking, Fishing, Pottery

    Introduction: My name is Duane Harber, I am a modern, clever, handsome, fair, agreeable, inexpensive, beautiful person who loves writing and wants to share my knowledge and understanding with you.