The debate around the use of LLMs in research is loud, and it is getting louder. Accountability. Hallucinations. Made-up references. Confident, unsupported claims. You name it. None of these issues are new. They are just amplified now that the bar for producing text has dropped to near zero.
So the question I’m asking myself is: how do we take advantage of LLMs without compromising our learning? Yes, we are writing papers faster than ever. But are we learning something during this shorter process? Yes, we can speed up PDF creation by 100x, but can we speed up our learning by, say, 1.5x? I dunno.
I also understand that LLMs are useful tools, and students should understand how to use them better. Here, I provide a few tips to help students use them while avoiding missing the learning part.
LLMs can accelerate your output by 10x. Maybe 100x. However, they can hardly accelerate your learning at all. Maybe they even decrease your learning pace. Cognitive debt is a real thing. Learning is still slow, still painful, still made of confusion and wrong turns and the occasional moment when something clicks.
The shortcut is right there. You can paste a paper into a chat window and get a summary in three seconds. You can ask for an explanation of a concept and get something that sounds right. You will save hours. You will also learn nothing.
I’m not exaggerating. There is no version of “I read the LLM summary” that gives you what reading the paper gives you. As we found in a Reddit post: “You can’t outsource the struggle and still get the skill”.
The rule I’d give every student: don’t outsource anything you need to learn. Use the LLM after you understand something, not before. Use it to check your work, not to replace it.
LLMs are trained on text. They produce text. But they are not the best source of text. Although LLMs can follow strict grammar rules, their vocabulary mimics the average one. And as we all know, averages aren’t that good.
Since you are using an LLM, you must always read its output. However, do not restrict your literature to the LLM’s output. People out there are already talking like an LLM. Don’t be an LLM. Improve your vocabulary.
Read good research papers. Read good blogs. Read good novels. Read things outside your research topic. Get used to reading long documents. Get used to reading every day. Read about topics that you love until you love to read.
Research is, at the end of the day, a deeply human activity. You’re trying to understand and explain something that nobody else has understood. This requires enormous effort. The more you read about how humans think, fail, and rationalize, the better you’ll get at it.
In the same vein, talk to real people. I know it is easier to ask questions to an LLM, but do not forget to ask questions to real people. Express your half-baked ideas. Half-baked ideas are how baked ideas start. Since we are very connected with these tools, we may miss how to connect and express ourselves with real people.
Using an LLM is fine. I use LLMs all the time. However, read and question every word they write.
A common misuse of LLMs these days is to let them create non-existing references. The LLM cites a paper with complete confidence: author, year, journal, title. And the paper doesn’t exist. Or the opposite: the paper does exist, but the LLM says something different from what the paper claims.
Please, please check the references. Every one of them.
And don’t stop there. Read every word the LLM produced before you put your name on it. Not because the LLM is malicious. It isn’t. But because it is a very confident guesser. A confident guess can be helpful. An unchecked confident guess in a paper with your name on it is something else entirely.
You are the author, not the LLM. The LLM is a tool. Tools don’t have reputations to protect. You do.
Here is where LLMs genuinely shine, and where I think students underuse them.
These tasks have low stakes. They consume real time and mostly create boilerplate work. Offloading them is a big win. But there is a catch. You have to know how to do these things without the LLM. One Reddit user put it nicely:
“If you don’t fundamentally understand what you’re programming at a basic level, and the code itself, to verify if it’s doing what the method should do, you’ll almost certainly make mistakes.”
This applies far beyond code. Automate the routine, but keep the reasoning to yourself. The moment you start automating the reasoning, you’re no longer doing creative work.
I’ve seen students reading LLM-generated answers out loud in meetings, almost word for word, without processing them in the first place. I’ve seen classwork that sounds like nobody wrote it: fluent, confident, yet shallow. I’ve seen people in conversations type a question into a chat window and read the answer back as if it were their own thought.
Think for yourself. Speak for yourself. Please.
Given the whole body of knowledge in the universe, it is obvious that we are far from knowing a good chunk of it. Actually, what we know is that we don’t know. Saying “I don’t know” is not a weakness. It is fine. It is expected. Indeed, this is where every real investigation starts.
Don’t use an LLM to cover what you don’t know. When you repeat an answer you cannot explain, you are not really hiding the gap. You are making it visible.
A better move is to acknowledge what you don’t know and use that as the starting point for learning. Ask the LLM to help you map the gap, suggest readings, test your understanding, or challenge your reasoning. But don’t present its output as if it were your own knowledge. Only you can be you. Don’t try to be an LLM.
None of this is an argument against LLMs. I use them every day. They’re genuinely useful, and refusing to engage with them is, at this point, a form of self-sabotage.
But the students who will thrive are not the ones who use LLMs the most. They’re the ones who acknowledge their limitations and know when not to use them. They will be the ones who still spend a good amount of time reading a paper, writing a paragraph from scratch, sitting with confusion until it resolves, and admitting they don’t know something.
]]>Back then, the tech environment was completely different. The way you applied for jobs, the onboarding process (or the absence thereof), the work environment, the benefits, the compensation, the cloud infrastructure — everything was different. AWS wasn’t really a thing yet, so it was common to deploy services in-house, and continuous integration and delivery were far from being standard practices. People were selling agile left and right, but waterfall was still alive and well. Ruby on Rails — and other frameworks that inherited its architectural and design choices — were mostly discussed in startups, while mature companies still leaned on JEE technologies. That was the scenario I left behind when I transitioned into academia full-time. Thinking and writing about it today, it feels like a lifetime ago. And it really is.
In 2021, when I signed with Zup, AWS was already the king, with dozens (hundreds?) of different services, some of them similar but still subtly different. People seemed to know when to use what, and why. AWS also had company, such as Azure and Google Cloud, though AWS remained the default choice. Java was still widely used, but other JVM-based languages, such as Kotlin, had carved out their own space. Python — which until then I had used mostly for fun — was being adopted far beyond its well-known statistics and data-science niche. JavaScript had grown into an entire ecosystem of its own. And finding a job had changed just as much: LinkedIn had become the king of career moves, and the selection process often started with someone reading what you had written there. Talent acquisition (TA) folks would find you and reach out. Then came several rounds of interviews — technical, non-technical, behavioral — until, with some luck, an offer arrived.
Nothing new here; today, people talk about tech careers all the time. But in 2021, all of this felt strange to me. There are a few things I wish I had known as a researcher before joining a tech company. This post is for my 2021 self.
Work hard to build a good relationship with your manager. I had the privilege of working with great managers at Zup, but I know this is not always the case. A great manager can shape your career, put you on challenges where you will shine, step in when things start going sideways (and fix them before they escalate), and see you as a partner rather than just someone who reports to them. Great leadership is perhaps one of the most important things you can have in a job. Forget about all those benefits, days off, and fancy offsites. Only a great manager can give you clarity about your work and your future at the company. This is particularly important for academics, who, in their everyday life, do not really have a boss.
Get to know AWS. As I said, AWS is the king today. But when we run academic experiments, we rarely go as far as publishing them on a cloud provider. Maybe today, with AI research, we buy (or receive) a few credits to run experiments in the cloud — but that’s not the same as deploying and operating an application there. When you deploy, you need to think about permissions, security, roles, storage, processing power, deployment strategies, and, of course, budget. Combining all these services requires knowing which ones fit your scenario and — perhaps more importantly — which ones fit your company’s compliance requirements. Take your time to get comfortable with AWS.
Practice system design interviews. The selection process today is full of interview stages. One of the most famous is the system design interview. You will be asked to draw, on a digital whiteboard, the architecture of a system, often related to the domain of the company you are interviewing for. So, first, take the time to understand what the company actually does. Second, read about the main components of modern systems: HTTP (and its patterns), caching (providers, what to cache, when to invalidate it), databases (relational, NoSQL, how to structure them, when to shard or split), queues (and when async messaging makes sense). Think about how these pieces connect. Assume scalability is always a requirement, so practice adding nodes to your design. Think about failure modes — what is the single point of failure, and how do you recover? Default to asynchronous communication. Read a good book or two on the topic and practice by sketching a few designs on your own. Researchers usually have a solid theoretical grasp of system design, but this kind of hands-on practice requires you to understand how these tools actually work — and their trade-offs.
Practice writing code. Here, I would tell my 2021 self to write fewer papers and write more code. The coding challenges we tackle for papers and research tools are, in my experience, on par with what people build in industry. The main difference is maintainability. In academia, it is very common to write code that exists only to support one particular publication. After the paper is out, the code is effectively dead. In industry, you rarely write code with a one-month shelf life (it happens, but it’s not the goal). The goal is to have your code used for years, which means that maintainability and testing become first-class concerns. Read about good practices, and do not skip tests — even a simple coverage metric is more than enough to get started.
Learn to operate what you build. In academia, once you finish an experiment, you move on. In industry, once you ship, the system is yours to keep running. That means getting comfortable with observability: logs, metrics, traces, dashboards, and alerts. It means learning to debug a misbehaving service in production at 10pm, before the customer complains. It also means understanding deployment pipelines, feature flags, and rollback strategies. For a researcher, this is a mindset shift: the work is not done when the code is merged — it is done when it behaves well for real users, under real load, for a long time.
Invest in written communication. In industry, especially in remote setups (which was my case), most of your impact flows through writing: pull request descriptions, design docs, incident postmortems, roadmap proposals. Unlike academic writing, which optimizes for rigor and completeness, industry writing optimizes for clarity and decision-making. A short, well-structured document that helps a team make a decision is worth more than a perfectly argued 20-page text that nobody reads. Learn to write for busy readers. For academics, this is both easier than it sounds (we already know how to write) and harder than it looks (we have to unlearn some of our academic habits).
Move fast — but remember that more speed is not always the answer. A couple of years ago, I wrote about the dichotomy between rigor and value in industrial research. In the lab, time is our friend: an experiment can take six months, and if the results feel off, we rerun it. In industry, time is the scarcest resource. PoCs are expected to be short-lived — you build something small, you ship it, you see what happens, and you discard it without ceremony if it does not work. “Fail fast” is not a slogan; it is the operating model. And, very often, velocity wins the game over quality: a rough version that ships this month beats a polished one that ships next quarter, because by next quarter the problem has already moved. The AI era only sharpened this trade-off — code is now generated faster than we can review, and the pressure to keep shipping only grows. Honestly, I am still not sure what the actionable takeaway is here. I do not believe that more and more speed is always the answer. But I also know that the academic reflex to “do it right” can paralyze you in a context where “right” is a moving target. Somewhere between these two poles lives the job of anyone moving from academia to industry.
Five years in, I realize this letter is less about the specifics — AWS, LinkedIn, PoCs — and more about a shift in posture. The academic in me is still there: I still read papers, still run experiments, still write pieces nobody asked for (like this blog post). But the industry taught me to hold my conclusions more loosely, to ship before I feel ready, and to treat most decisions as reversible ones.
]]>I’ve been practicing jiu-jitsu for more than 10 years. One might beleive I’m a skilled athlete. However, I’m not the top black belt in my weight division, not in my neighborhood, and not even in the gym where I train. That’s fine. What matters is the path and what I learned and pushed through along the way.
For a long time, I thought I would never tie that belt. Jiu-jitsu was never a priority, and as you approach your 40s, life seem to accelerate. Managing training alongside a family with three kids, two jobs, and events such as COVID, injuries, and everything else makes anything outside your critical path harder to keep going.
During part of my PhD, I also believed I would not be able to complete it. Somehow, I did.
Looking in the mirror, these journeys might appear unrelated — one in libraries and struggling with peer-review; the other on the mats, between chokes, sweeps, and being smashed by (often smaller) training partners.
But the deeper I think about it, the more I realize that earning a PhD and earning a black belt are essentially two sides of the same coin.
In both a PhD and in jiu-jitsu, progress rarely feels real while you’re living through it. You show up every day, struggle through complex papers or difficult drills, and feel stuck. Entire months may pass where nothing seems to move, even though you’re pouring effort into the process. I quite remember how hard it was to read and fully understand a few papers on my field. I invested time and effort, but nothing moved forward in the research itself. This lack of visible progress made me fell bad, especially when others appear to be advancing faster than I.
But the truth is that growth happens in the micro. You don’t notice your reasoning sharpening until you reread your early drafts, and think, “oh, how could I write such a bad piece?”. On the mats, you don’t realize your timing is improving until one day you land a sweep to someone who usually gives you trouble. When someone says “good job” — for a paragraph that you wrote or for a technique you performed — they’re reacting to a visible result of many repetitions that stayed unseen.
Plateaus, regressions, sudden breakthroughs — all of these are signs that progress is happening under the surface. Whether you’re fighting with theory or with an opponent, you’re constantly building layers of understanding; they will only make sense over time.
Failure is not an accident in either path — it’s the design.
In academia, you are “submitted” by harsh comments, rejected papers, failed experiments, and theoretical dead ends. Reviewer #2 (a joke that refers to someone whose job is to reject your work) might not use an armbar, but the effect of rejecting your deared paper is similar; pain is part of the game 🙂
On the mats, the lessons are more literal. You tap (the signal that the game should stop), again and again, to people who are more technical or simply having a better day. Every submission exposes a gap in your understanding. The mat delivers immediate, honest, and often uncomfortable feedback.
Both worlds teach resilience through repeated defeat.
Reviewer comments push ideas forward; a choke shows where you left space. Over time, you learn to welcome these small issues as part of the process. Every researcher had several paper rejected; Every jiu-jitsu practitioner had tapped hundred of times. Eventually, rejections would not hurt that much anymore. You learn to reset, adjust, and move on. Perhaps your growth is measured by how many times you’re willing to get back on track.
Motivation comes and goes. If your work depends on motivation, progress stalls.
The real progress happens on cloudy and raining days, when you don’t feel inspired, energized, or confident.
When you are willing to get back to that manuscript that you have been working for 2 years non-stop. You want to see the devil himself but you don’t want to look again at that manuscript; but you do it nevertheless. Or when you force yourself to go training despite fatigue or frustration. Showing up can be the best thing you can do.
Hopefully you become someone who doesn’t depend on emotional peaks to stay committed. You work because the work needs to be done, and no one else will do it for you.
From the outside, a PhD defense looks like the end of a journey. For anyone who went through it, it marks the start of working as an independent researcher.
You leave knowing a narrow area well, while also realizing how much remains unknown. You may know a couple of research methods – I myself was very familiar with performance experiments. However, the set of tools available out there is richer, and the contexts where they apply vary a lot, which requires a deeper understand of the trade-offs and the conner cases. During the PhD, you rely on an advisor. After that moment, you are on your own. Mentors and colleagues can help with questions, but the direction and the final say will always be yours.
The black belt in jiu-jitsu represents a similar paradox. Many people might assume you have mastered the art. However, your colleagues at the gym know it means you’ve only mastered the fundamentals. You may understand the main positions, transitions, and submissions. I think I was very good in finding and doing ambars from different poistions. However, details remain endless.
Another example: you shouldn’t use the same techniques when fighting someone around your weight and someone weighing 50 kg more than you. Similarly, you need to adapt your game when rolling with a blue belt in their early 20s, full of energy, or with someone who has been a black belt for 30 years. This adaptation is something you have to understand, adjust, and often figure out by yourself. You can have colleagues and mentors, but the direction is still yours.
Getting a PhD (or a black belt) isn’t the finish line. It’s the moment when you realize how much there is still to learn; thus it’s also when refinement really starts. The responsibility only grows, not shrinks. Someone in the mat is now trying to do the same techniques in the way you do. Someone who read your papers is trying to perform the same experiment.
Both milestones signal readiness, not completion (which may not even exist). They mark the moment when you’ve built enough foundation to explore your craft with depth, to contribute to your community, and to pursue mastery with intention.
They also remind you that the learning process should never stop.
Remember that your toolbox is as rich as the professional using it. If all you have is a hammer, everything looks like a nail. Learning to use a hammer matters, but you can only move to more complex problems if you go deeper into the details.
The end is simply an invitation to begin again — at a higher level.
]]>I just figured it out as I was driving back home after the Carnival break, while the kids were sleeping in the back seat, and the silence inside the car allowed my mind to wander far and wide.
I remembered my slide presentation, the time it took to refine it, the presentation itself, the committe (composed of a few of my academic idols (such as Paulo Borba, Marco Tulio Valente, Marcelo d’Amorim and Fernando Quintão), a few questions of the committee, my advisor Fernando Castor with his usual morning cold, and my family in the audience..
Although the committee was well-prepared with thought-provoking questions (Quintão even asked if he could use the whiteboard to write down some equations he was thinking about), overall, I’d say that morning felt more like a celebration than a defense itself. I really enjoyed discussing open questions, hidden problems, and potential ideas for future work.
Then I remembered why I started doing research.
Indeed, experiencing the depth of those discussions was something that attracted me to research. During my professional experiences before my PhD, I remember that most discussions were shallow, decisions were made by whoever was the most convincing, and very few people questioned them.
On the other hand, in the research seminars I attended, I discovered that questioning is common and very much welcomed. Actually, it is the researcher’s job to question everything, all the time. By that time, I already knew that is no such thing as an absolute truth. What I didn’t know is that we can and should question anything that seems like the truth. We walk on a quicksand floor of not-extensively-questioned-yet truths.
I then learned that the questions are more important than the answers.
I also learned that there are methods and tools to confront questions.
If you use one of these tools to question an important belief, you maybe can write a scientific paper about it. Imagine yourself, an young boy from the countryside of Amazon rainforest, writing a scientific piece, explaining that what people think is true, may not hold under certain circumstances. In a perfect word, it would be analogous of Paul Mccartney writing Yesterday. All my problems seem so far away.. 🎶
This was all new to me, and I was super impressed by all of it.
I then started to have my own papers being accepted for publication. This raised me emotions that I hardly felt in a work environment. Being able to present my own ideas, to world experts, receiving feedback, sometimes even congratulations, was so intellectually stimulating. Conversely, having my ideas rejected was like a punch in the face, which really hurt me. Unlike other kind of work, in this academic work I felt many different emotions.
I was so grateful to be part of it, and I want to participate more, much more.
Then something happened.
Fast-forward a few years down the road, and I found myself caught in the academic trap: writing as many papers as I could, accepting every invitation to any kind of academic service, advising dozens of students, applying for every possible form of funding, and trying to do a decent job at teaching — just like any other researcher out there.
At some point, reviewing papers was no longer enjoyable, and I started thinking about declining service invitations. But how could I decline a review invitation if I planned to submit papers to the same conference? The community remains healthy only if everyone fulfills their duties. Let’s try our best to push through.
By writing a lot, I also discovered that there is a writing algorithm. The more I wrote, the more I observed common patterns in scientific reports. On the one hand, understanding these patterns helped me structure papers in the expected “shape.” On the other hand, following these structures stripped away all the glamour and creativity from writing. There is nothing exciting about writing papers. Paul McCartney, you are unique.
Moreover, since a large portion of my work was devoted to writing, I gradually stopped doing things like programming. That wouldn’t have been a big deal — except for the fact that a significant part of my research agenda was programming-related. How could I conduct programming-related research if I no longer program myself? Sure, we can always do user studies on programming languages and frameworks, but this started to feel strange to me.
And suddenly, traveling became a burden. In the beginning, traveling was exciting and fun. Meeting new people, having drinks and laughs with old friends — it was always nice. But eventually, I found myself visiting cities where the only things I remembered were the airport, the conference hotel, and the closest restaurant. For those who don’t know, traveling to a conference is an immersive and mentally exhausting experience. Activities start in the morning and go on until the end of the day. If you’re also under time and budget pressure, you may not have the chance to enjoy what else the city has to offer — which was often my case.
Finally, I realized that academia is more like a game. And like any game, it has rules — rules that, in this case, revolve around metrics. What’s frustrating is that many people seem more interested in playing by the rules than in the actual discussions and discoveries. In the end, it’s not about uncovering the unknown; it’s about mastering the system.
In the beginning of my career, I had a more romantic view of research, but that faded over the decade.
Now, I see research purely as work — just another task I have to do, whether I like it or not. Obviously, there are still research topics I enjoy working on, but that doesn’t happen as often.
Research is not a toy for adults to play with; it’s a job with its ups and downs. But it took me 10 years to understand that.
]]>In my first year, I was working as a researcher in an educational department. Part of my job was to uncover potential problems the engineering teams had and help them overcome these challenges. We conducted several research experiments, including tool building, interviews, surveys, and so on. Some of these results were published as research papers, others on the company blog, and some were shared through internal meetings.
Things were going well, but in 2023, tools like ChatGPT and Co-Pilot became an unquestionable reality in the software engineering arena.
Although these tools are great, they hardly provide a definitive answer to your prompt, mainly because they were not trained on your business data. For instance, if you ask, “write a wrapper for my client’s API,” Co-Pilot can’t help much other than writing a generic wrapper that would require substantial editing from a developer before being pushed to a Git repository.
As a consequence, with the help of these tools, a new market opportunity opened up for specialized coding assistance that could provide more contextualized information for developers. These coding assistants use information retrieval techniques to mine proprietary company information to enhance an LLM’s prompt. By enriching these prompts with contextualized information, these tools improve the odds of the output being more enriched and contextualized as well.
Given this new landscape and my academic background, I was invited to join a newly created team that focused on building such a specialized coding assistant tool.
Although I’m not that bad at designing programming solutions, I have to admit that I was not well familiar with the current libraries, frameworks, and infrastructure tools. Therefore, the first weeks of this move required a lot of patience from my team, helping me set up things correctly.
Some of my initial tasks involved experimentation, which was good given my background. For instance, since we are enriching prompts, what kind of information should we prioritize in the prompt? In what format should this information be provided? How long should this information be or last? These questions are all interesting, do not have clear answers (thus require experimentation), and could directly impact the tool’s output. Some of these experiments became pull requests that were integrated into the product’s repository. This was great.
However, when building a product, there are many other tasks aside from these experimentation-intensive ones. Therefore, I also took charge of some engineering-intensive tasks.
While learning all these libraries, frameworks, and tools, my research-oriented mind often contemplates other research studies that I could conduct. There are dozens of research problems that are worth investigating, some of which could impact our product.
Unfortunately, I will not tackle most (perhaps all) of these problems using an academic lens. Instead, I may approach them using an engineering lens.
This means that, as I mentioned in the previous blog post, time is key here. Even if we end up creating a robust methodology and observe interesting findings, we may still not have the time to document the context and findings in a beautifully written academic paper.
Building this understanding of problems that are worth exploring is a great asset. In academic research, it is common to believe that any problem is worth investigating. However, in industry, we may have to discard ideas that do not contribute to the business’s goals. It is, though, relatively easy to figure out the ideas that contribute to the business goal; for instance, those that make the clients happy.
In research, it is a bit harder to find the ideas that contribute to the business goal. After all, what is the business? Who is the client?
This is not to say that every research paper must need a business/client context. Indeed, pure research is important and often leads to significant breakthroughs.
However, fields like software engineering are, by design, less pure science. In such fields, if recent advances do not consider the intricacies of human interaction and their relation with the developers’ tools, such advances would rarely translate into features used in real-world development tools.
This is the scenario that Masad calls “metagame”, which happens when we focus too much on the theoretical aspects of a subject, without considering the underlying context. He goes even further by suggesting that: “Eventually, the lack of contact with reality will corrupt the field.”
I found this text while doing a review for ICSE 2023. During my review duties, I remember reading some papers that, although very well-written, well-justified, and well-performed, I found them unrealistic; or, adapting Masad’s term, too “meta”, perhaps a “metaresearch”.
This last week, I also attended a Brazilian software engineering conference. While there, I found some of the presented research, again, very much meta.
However, the most intriguing thing for me is the fact that I have been doing peer-review for quite some time, I have also attended dozens of software engineering conferences in the past. But the metagame only became clear to me when I started working for an engineering team.
]]>Undoubtedly, writing is a fundamental skill we acquire during our master’s and Ph.D., as it could contribute to our ability to reason about things and to communicate effectively.
Researching is paramount when you do not know how to solve a problem, while discovering innovative solutions.
Curiosity and perseverance are also well-known as key traits necessary for delving deeper into topics and meeting paper deadlines.
However, I want to focus on a skill that tends to be overlooked, one that I had taken for granted until now — the ability to sit down and study.
The never-ending need for continuous learning in the software tech industry may not come as a surprise.
Each day brings both mundane and extraordinary advancements, from the introduction of new libraries and frameworks to groundbreaking technologies that reshape entire fields, such as ChatGPT.
Staying up-to-date with these developments is crucial.
Regardless of what you want to keep track, the point is that it is important to develop a process that allow you, along with all your daily demands, to learn continuously.
As my mother often said when I was a teenager:
“you just have to sit down and study”
Unfortunately, the “just” in the above sentence is a bit harsh, because it suggests that sitting and studying is an easy task.
In fact, studying can be quite challenging, in particular because it involves several other skills and abilities.
It involves grabbing loads of information, sometimes convoluted, in an attempt to fit them all into our minds. We encounter sentences where we recognize each word, yet struggle to grasp the overall meaning.
It requires patience as we read and re-read until clarity emerges. It demands focus when faced with overwhelming subject complexities.
Studying goes beyond mere memorization; it requires critical thinking to analyze information from diverse sources and synthesize it coherently.
Unfortunately, in the era of TikTok and endless scroll, studying becomes a battle against all these temptations.
Study is hard work.
The research journey, in particular, forces us to embrace long hours of solitary work.
Translating an idea into a well-researched publication demands weeks, months, and perhaps even years of dedicated and deep study.
Pursuing a master’s or Ph.D. — if fortunate enough to have the opportunity to do so full-time — provides a unique window of freedom to devote substantial time to studying. It is during this educational journey that many researchers establish their studying routines.
Of course, those who haven’t pursued formal education can also train themselves to become exceptional learners. However, the research process inherently enforces and strengthens this ability. As Richard Feynman once said, “I was an ordinary person who study hard.”
Interestingly, despite the advancements in technology like ChatGPT and its counterparts, our studying process still closely resembles that of scholars who conducted research long before the age of computers and the internet.
Studying remains a process that demands time, focus, and discipline.
However, it is crucial to note that studying today may be even more challenging due to the constant interruptions we face. The overwhelming number of distractions can make it difficult to find dedicated study time.
Nevertheless, like riding a bicycle, the ability to study is a skill that endures.
While the dynamics may differ in other disciplines, the tech industry presents a tough challenge when it comes to devoting uninterrupted weeks or months to study.
Its fast-paced nature demands collaboration and constant interactions, leaving little room for prolonged periods of personal study.
Yet, the pressing need to stay updated with the latest advancements in the field is undeniable. Software practitioners must absorb and master new knowledge while maintaining the current workflow.
In a rapidly changing environment, the ability to study complex topics and master new skills becomes a significant advantage for one’s professional growth. But the process of studying is still hard.
My final and personal take here is: for academics, it might be easier to start studying.
I’m not saying that those with an academic background do not procrastinate or so. But it seems that the barrier to literally sit down, open a book, and start reading is lower.
In a competitive, fast-paced environment such as the tech industry, this is a great skill.
Are you in the tech industry and looking to broaden your knowledge? If so, the ML4SE (Machine Learning for Software Engineering) newsletter might interest you.
In this newsletter, written in Portuguese, delves into how developers can leverage machine learning techniques and models to enhance their work.
Join us if this sounds fun!
]]>I remember discussing our research questions with dozens of students at all possible levels. Here and there, I noticed that some students were more interested in building things than thinking about the methodological stuff, which I now understand. Building stuff is something concrete; they can see something happening, share progress, and see their learnings taking shape. But, on the other hand, the methodology seems more abstract and complicated to reason about.
But the methodology and, in particular, the research questions drive the direction of our work. The research question brings some light on what we should search for. Consequently, it only makes sense to build things if we know what we should build. But how do you know what you should create? How do you know if it makes sense to build that particular tool you are building?
At some point, I started to ask this question to students:
Which one do you think is the most important: the question or the answer?
The answers varied a lot, which was something intriguing to me.
For a moment, think that you are traveling for a random conference at a random place. At the conference, people seem to be friendly and easygoing; you are alone, and you want to make some friends.
Let’s say you want to approach someone but don’t know what to ask to start a conversation.
Perhaps you approach someone with a broad question, like “hey, how are you?”. This question could confuse your potential friend if you don’t know the person in advance. What kind of answer are you expecting? Do you want to know about her day? About her work? About her travel? Yes, I know; questions like this could break the ice. But these general questions — with little context — can also make it harder to engage with people. Without context, such questions can also sound a bit awkward.
A bit of rephrasing and context would improve this question a lot, though. For instance: “how was your day at the conference?”, “what was the best presentation you saw today?”, “what are the subjects you expect to learn more about during this conference?”.
Note that these questions are more specific because they have more context. With this context, the asker drives the answer to a more specific point. Thus, the respondent can better reason about the question and provide a more thoughtful response. Note that these questions do not assume that the asker knows anything about the respondent. Even better questions could be made if the asker knows any information beforehand.
On the other hand, if the question has too much context, the answers could also be of little value. For example, if you approach someone and ask if she liked the third example of the 50m keynote talk that happened yesterday afternoon, what could your colleague answer? Are you sure that she watched the keynote after all? Did she like the keynote so much that she could recall the order of the examples? Even if she can remember that particular third example, what is she supposed to answer? “Yes, I liked/ No, I didn’t”?
Perhaps the first conclusion we can draw from this fictitious example is that questions that are too general or too specific can lead to misleading answers.
Once again, context is king. Without context, your question becomes vague. With too much context, your question becomes hard to reason about.
But this is not the most important conclusion.
This is the most important observation.
Good questions tend to lead to good answers.
Similarly, bad questions tend to lead to bad answers.
If the question is poorly formulated, the answer does not matter.
Why is 42 the answer for the meaning of life, the universe, and everything?
How could a profound and thought-provoking question like this be answered with just a number, without any further explanation? What can we learn from this?
Easy.
Because the question is vague. What are we supposed to answer? What part of life, the universe, and everything else should we reason about? Is this question about my life? My family life? The history of life on earth? What do I even know about the universe? And, by the way, what do you mean by “everything” else?
This question is poorly formulated. It is broad, and it does not provide enough context.
So, why is 42 the answer? Because it does not matter. The question does not make sense, and neither does the answer.
When students want to build stuff, they want to work on the answers. But we can only build stuff if we know what we should build, if we know the question. Without understanding the question, students may end up building stuff of little value or something already made in the past (which we were unaware of).
That is, students want to work on the answer — which is the solution — without understanding the question — which is the problem.
If we start the research thinking about the solution, we may lose track of the problem. This, in turn, could lead us to answer things like the “42” above. 42 is obviously an answer, but an answer that does not add much value.
I often encourage students to bring their problems to the research program. Often, students mention that they want to build the next refactoring tool that would remove that bad smell that happens in every other commit in their company codebase. Sounds cool and fancy, doesn’t it?
But note that, in this particular example, the student brought a solution, not a problem. Building a refactoring tool is a solution to a problem that is, at this moment, unknown. As some people like to say, “it is a solution looking for a problem.” Does it make sense to have an answer looking for a question? Maybe not.
Curiously, thinking about the problem (and the question) seems to be a challenging exercise. It requires a lot of understanding of the research method, the literature, and the subject under investigation. However, it also requires discipline (to ask new questions constantly), curiosity (to keep on learning and digging deeper), and courage (to receive criticism on half-baked ideas).
So far, we have understood that question drives the answers, and bad questions lead to bad answers, but good questions are challenging to come up with.
How to design a good question? I don’t know. I don’t have an algorithm.
I may need to think more about it, perhaps for another blog post.
]]>My job at Zup is a mix of many things. A bit of engineering, a lot of management, a bit of writing, a lot of meetings, a bit of paperwork, and (sometimes a bit, sometimes a lot of) research. Even though I had some experiences collaborating with companies while I was a full-time professor, this was the first time that one of my main goals was to impact software engineering teams through research. Despite my academic background, this was not something crystal clear to me.
During this year, I’ve talked to many engineers/directors about how we could bring value to them. The more I talked to people, the more I understood that the research we did in the lab would hardly be translated to the company. Many variables impact this process (such as the scale of the experiments, the quality of the codebase, etc), but I believe the most critical variable here is time.
In the lab, we usually are OK to perform one experiment in, say, 2, 4, or 6 months. It takes time to understand the problem, train the students, set up the experiment, etc. It’s not uncommon to have studies going on for over a year, sometimes more. If the student leaves the lab, it’s OK to find and train a new student, which would take a few more months. Even in the best circumstances, if we believe that the results are not just right, it’s OK to redo the experiments, in order to make sure that what we see indeed makes sense. In the academic lab, time is our friend. Obviously, students have to graduate, projects need to be finished, and papers have to be written. But, generally speaking, it is OK to spend more time reflecting and refining the research before making it public. You know, people may make decisions based on our research, so we should value the rigor of our methodologies.
On the other hand, engineering teams operate at a fast pace. They ship code regularly, perhaps every day. They provide value to their clients through new features. Sometimes the value of a new feature is not always clear, so the teams have to experiment a bit. Experimentation requires early feedback, which is often much more appreciated than waiting for months to approach the clients. Teams have to move fast; otherwise, other teams will. And no problem if a new release comes with a bug. We can fix it in a minute and ship it again. Therefore, mottos like “move fast and break things” are not just an anecdote. In engineering teams, moving fast is often more desirable than having a bug-free product.
This is the dichotomy of industrial research.
While researchers favor methodological rigor, engineers favor delivering value fast to their clients.
In this context, it is not feasible to believe that we could just apply our favorite academic method in the industrial context. This will not work because the two teams operate on a different schedule.
If I tell to engineering teams that I could help them in understanding this particular problem they are facing, but it would take, say, 3 months, chances are they will look for the solution themselves. Depending on the context, in 3 months, maybe the whole project had changed its direction, and that particular problem may not relevant anymore.
By design, things like systematic literature reviews are just unfeasible to do. Creating that tool that will improve the precision and recall of state of the art might also not sound like an appropriate thing to do.
Not only it is not feasible to apply certain kinds of research methods, we also believe that we should relax parts of the methodological steps. For instance, instead of transcribing and coding dozens of interviews, we may conduct just a few ones, skip the transcription part, and report just a few insights of what we have heard here and there. Similarly, instead of writing one new tool from scratch, we may use (or adapt) an existing academic tool to the company’s context.
We can help the teams with their needed velocity by relaxing the research method. However, by relaxing the research method, our research becomes less likely to be well-received in the academic community. And it’s our interest to contribute with scientific knowledge to our community.
This is the dichotomy of industrial research.
At this point, I realized that our research team might be unable to contribute to the research community with new methods, techniques, and tools, just because they take too long to be polished.
And that’s OK.
I also realized that we have a unique opportunity to explore the behaviors of the engineering teams during the software development process. Since we are near to the teams, we can ask questions like: What are the team’s perceptions when adopting a research-oriented tool? To what extent our relaxed research methods could influence teams in their decision-making process? How do the teams find and share knowledge? And the like.
In summary, I learned that industrial research requires its own set of methodologies and tools. It is not feasible to translate our full-fledged academic methods to the industry in the hope they will work. Instead, we should tailor the methodologies and questions to the practitioners’ needs. Although writing this last sentence seems something obvious now, it was not evident in the first place 😊.
In an academic environment, most questions deserve investigation. In an industrial setting, just a subset of questions deserves investigation.
And perhaps this is the beauty of industry research.
During this first year, we made a few submissions, and one paper was accepted. For the next year, our plan is to improve our hability to conduct and share industrial research. Let’s see how far we can go.
BTW, if you are a research looking for an industry pattern to perform your experiment, drop me a line; we can make things happen at Zup!
There is much more to say about this first year at Zup, about other things I learned during this small time window. I also miss a bit writting blog posts. I really hope to keep this blog more active in the near future.
]]>Em 4 de Março de 2018, Shriram Krishnamurthi, um professor da universidade de Brown, nos EUA, escreveu uma thread no twitter desmistificando a fantasia que é estender um deadline, descrevendo o impacto que isso pode causar aos autores.
Sempre que vejo alguém próximo comentando sobre as vantagens de adiar um deadline, eu revisito esta thread pra me lembrar dos argumentos do professor Shriram. Sempre que releio, reforço minha crença de que extensões de deadline não ajudam; na realidade, são prejudiciais para comunidade.
Como eu não tenho o discernimento nem eloquência do professor Shriram, pedia a sua autorização (que gentilmente me concedeu) para traduzir, na integra, o texto da thread.
Espero que o texto ajude a coordenadores de comitês de programa de eventos de pesquisa a refletir e, eventualmente, ajudar a mudar o status quo dos eventos nacionais; estes, em grande número, baseados em extensão deadlines.
Suponha que eu tenha três alunos A, B, C com deadlines em semanas sucessivas. Eu aloco tempo necessário para finalizar seus artigos nas últimas semanas.
Agora o deadline de A foi estendido em uma semana. O novo deadline agora está competindo com o tempo que eu prometi a B. Tenho duas escolhas, e nenhuma delas é satisfatórias.
Obviamente, os concorrentes de A que disputam os poucos slots não estão parados, então o artigo de A, de certa forma, fica menos competitivo em relação aos demais trabalhos que estão sendo avaliados (encare isso, todos os comitês de programa acabam tomando decisões relativas, especialmente em eventos pequenos).
Enquanto isso, nós esquecemos do aluno C. Mas até mesmo C sofre repercussões do deadline estendido. Se eu acabar dividindo meu tempo entre A e B, isso geralmente significa que eu tenho que trabalhar muito mais do que o normal, então quando chego para trabalhar no artigo de C, estou muito mais cansado e irritado.
ESSA NÃO É UMA SITUAÇÃO HIPOTÉTICA. Já vivenciei isto em várias ocasiões. Posso lhe dizer por experiência própria que isso É UMA MERDA. Eu nunca senti que foi me dado um “presente” e uma “oportunidade” para “melhorar” meu artigo.
Mas eu ainda nem cheguei no problema real. Você sabe quem fica mais machucado por isso a cada vez? Não foi A, B, C ou eu. Foi minha ESPOSA, que calmamente assumiu mais tarefas domésticas, e meu FILHO, que teve que se contentar com menos tempo com seu pai.
Você quer falar sobre tonar Ciência da Computação mais equitativa? Você quer falar sobre equilíbrio entre vida profissional e vida pessoal? Bem, mantenha sua palavra sobre os deadlines que você define. Conserte-os.
Se você não tiver opção e precisar mesmo mudar, não apresente como algo legal. Faça a coisa certa: DESCULPE-SE por aqueles cujo o horário você interrompa. É arrogância assumir que você conhece os horários de todos melhor do que eles mesmo, e que sua mudança é obviamente bem-vinda. Assuma que alguém pode não concordar.
Quando eu sou chair de comitê de programa, eu sempre digo que o prazo é FIXO. Eu digo isso em negrito na chamada de trabalhos. Eu falo sério. Esse é o meu CONTRATO com o mundo. Somo pessoas que lidam com software: entendemos o que significa honrar um contrato.
Agora vamos falar sobre estes tries alunos de pós-graduação. Será que eles não sabiam do evento? Será que eles não sabiam do deadline do evento? Será que seus orientadores sabiam? Eu acho que todos eles sabiam. Se é tão importante assim, por que não planejaram com antecedência?
Talvez o evento tenha um histórico de mudança de deadlines. Nós criamos essa bagunça e depois a agravamos. Isso privilegia aqueles que fazem parte da comunidade a mais tempo: eles sabem quando é o deadline “real”. Aquele que tenta entrar na comunidade recebe a má notícia. Isso é justo?
Resumo: extensão de deadlines não é puro bem. Eles são excludentes, eles perturbam vidas (literalmente), e criam trocas desagradáveis. Crie deadlines firmes e mantenha-os. Tudo fica literalmente melhor.
Mas se você PRECISAR estender um deadline, ao menos peça desculpas. Reconheça que você pode ter perturbado a vida de alguém. Considere marcar artigos atrasados como atrasados, de forma que os membros do comitê de programa possam levar isso em consideração. Faça disso um jogo mais justo. Fim do sermão (-:
]]>By collaborating, you not only divide and parallelize the work but, more importantly, it could expand our limited way of thinking. By bringing new collaborators to your research project, these collaborators could in turn bring different ideas and perspectives, that could lead to news questions and potential solutions. Moreover, there are a number of multi-institutions grants. You can only apply for those grants if you have stablished collaborations.
It is your job to create and foster a network of collaborators.
Before finding trying to find your collaborators, you should be seen as a trustable collaborator. In scientific work, this means that every data collected, processed, and analyzed have to be available for anyone interested.
We all know that when running to catch a deadline, code and data often become a mess. However, it’s better to have a mess that someone could use and clean in the future, rather than no code or data. It may be hard to find trustable colleagues if your data only exist on your computer or if no one has access to the coding scripts you wrote.
Sharing everything you produced during research work should be your motto.
More important than having dozens of collaborators is having a a small network. Think about your small network as those colleagues that you want to work in the long term, those you can trust.
Trust, in this case, means someone that could help you to achieve your goals. Think about broader goals; don’t think about papers accepted, think about advancing your carrear. Given the universe of researchers on your field, how do you know someone that could indeed help you to achieve your goals?
That’s though.
However, it may be easier if you think about researchers that work on the same “critical path’’ that you work. The critical path is “the path of work that is critical for their career advancement or fulfillment at the given moment in time’. Find someone that shares the same critical path. If you just get your Ph.D. and are on a Tenure Track job, try to find someone who are also in a TT position. Invite this person to join one or two research projects, and work really hard. After that, consider expanding your network. Ask if this new colleague has another colleague that she could invite for another project.
Grow your collaborations slowly, in particular if you don’t have students.
It might be tempting to invite one of the big names of your field to your network and do some joint work. It might not be a good idea, though. This because these experts might be too busy running their own research groups, along with many administrative tasks. That is, they are not on the same critical path that you are. Thus, they are less likely to actively participate in your research project.
Think about collaborators as those that could help you with your research project, but they don’t need to be on the same critical path as you. You also don’t need to collaborate with them in the long term. However, collaborators are critical to fill your knowledge/technical gaps.
While you are building your trustable network, keep one eye open for collaborators.
I believe there are at least three approaches one actively find new collaborators?
UPDATE: I wrote this text as part of a short e-book for young researchers. Given COVID outbreak, going to conferences is not feasible still in 2021. However, academic conferences are happening online. It is not the same thing, but there is still opportunity to find new colleagues. You just need to try harder.
This indirect approach is less systematic than the direct approach. The idea here is to keep one eye open for new collaborations while doing other things.
For instance, if you happen to have read a paper that was very interesting to you, why you don’t send an email to the authors congratulating them and mentioning what you liked the most? Similarly, if you are traveling (attending a conference, for instance), why don’t you stop at a nearby university and offer to give a talk? (obviously, it’s essential to get in touch weeks before with an eventual host). Still, if you have questions, instead asking the same colleagues or googling around, why don’t you ask the questions on social networks? The chances are that other researchers might have the same question or even know the answer.
The take-away here is that you could search for collaborators while not clearly search for collaborators.
In the same way that you are looking for new collaborators, other researchers are also looking for new collaborators. You can only be someone’s new collaborator if someone finds you.
If Google doesn’t find you, your new collaborators are even less likely to find you too.
Dedicate some time to create a professional webpage. There are dozens of cheap hosting services, many of with nice templates that one could easily edit in the browser. With a bit of hacking, one could run a personal website online for free. Regardless if you are paying for a hosting service or not, you should have an online (and up to date) website running. A simple website would contain your personal information (e.g., your name, where do you work/study, your research interests, and your email address).
Some researchers are worried about putting their plain text email address available since spammers might like it. But if you have any smart email client (e.g., GMail or Outlook), spam would not be an issue for you. Besides, you might also help other researchers to copy and paste your email address with ease. If you are still very concerned about this, consider using a simple HTML trick such as the following:
gpinto@ufpa.br
This <span> HTML tag does not appear for users when browsing your webpage while, at the same time, it hinders crawlers from finding an email address.
If you already have an up and running website, go one step further. Consider: