TSM Premium Survey

We are reviewing our Premium offering and considering where to take it in the future, with potential new features and benefits to enhance your gold-making experience inside and outside of the game.

Your feedback is important in shaping the future of TSM Premium and the direction we take will be guided by your input, so please take a few moments to share your thoughts and suggestions in the following survey – it is open to all TSM users whether you are currently a Premium member or not.

Please note you will need to be signed into a Google account to complete the survey, this account information is not shared with TSM. If the embedded survey does not load correctly, it can be accessed through the direct link:

https://docs.google.com/forms/d/e/1FAIpQLSc8_ftqHJqsejepvs6Lck7oyZuiO8l7q_r4ewWXRlGKdFwHAA/viewform

New Custom String Parsing

Introduction

Custom strings have long been at the heart of what makes TradeSkillMaster such a versatile and powerful tool for gold making. They allow the user to focus on their gold making strategy, and allow the addon to worry about doing all the complex math for them. For example, the default Auctioning operation settings ensure that items will be posted above their vendor sell price and a percentage of an average of the crafting cost, realm market value, and region market value, while transparently accommodating items that can’t be either vendored or crafted, all without requiring the user to ever need to manually calculate or input those values. In v4.13, we completely rewrote how the addon parses and evaluates custom strings to allow for much greater optimization and user debuggability. This blog post will dive deep into the changes that were made and the performance improvements they’ve enabled.

Background

TSM’s custom strings are defined by a set of functions and sources that can be used together in a single expression. As an example, let’s dissect the default Auctioning Operation minimum price setting:

check(
	first(crafting,dbmarket,dbregionmarketavg),
	max(
		0.25*avg(crafting,dbmarket,dbregionmarketavg),
		1.5*vendorsell
	)
)

There are a few things going on here. First of all, the outer check() function ensures that the first parameter is valid (and greater than 0); making the entire thing invalid if not. This is used here to ensure that the item has either a crafting cost, realm market value, or region market value. If all of these are missing for the item, we don’t have enough information available to properly post the item, so the custom string is made invalid to prevent posting the item at a potentially erroneous price. Assuming that check passes, we’re left with:

max(
	0.25*avg(crafting,dbmarket,dbregionmarketavg),
	1.5*vendorsell
)

The max() function here takes the higher of its two parameters. The first parameter is the average of the crafting cost, realm market value, and region market value multiplied by 0.25. The second parameter is 1.5 times the vendor sell price. This ensures that this custom string always evaluates to at least 1.5 times the vendor sell price, even if the market values / crafting cost is very low. Both avg() and max() also have a side-effect of silently ignoring any invalid parameters, meaning if the item doesn’t have a crafting cost or a vendor sell price, the custom string can still be evaluated based on the other parameters.

Let’s plug some values into this custom string and see how it gets evaluated. Let’s assume we’re trying to post “Spice Bread” which has the following prices:

  • Crafting Cost: 35c
  • Vendor Sell Price: 5c
  • Market Value: 12s 70c
  • Region Market Value: 11s 57c

If we simply plug these into our custom string, we get the following:

check(
	first(35c, 12s70c, 11s57c),
	max(
		0.25 * avg(35c, 12s70c, 11s57c),
		1.5 * 5c
	)
)

After evaluating the first() and avg() functions:

check(
	35c,
	max(0.25 * 8s21c, 1.5 * 5c)
)

The first parameter of check() is valid (greater than 0) so it simply evaluates to its second parameter, the max(), which calculated out results in 2s5c.

This is the same math that TSM is doing under the hood, but obviously TSM needs to do it in a well-defined and programmatic manner. It has traditionally accomplished this by relying on the fact that the custom string syntax (that is, the format used to write custom strings) is very similar to valid Lua code (the programming language addons are written in), and Lua allows for dynamically loading and executing code. In other words, we can treat the custom string as Lua code and execute it directly as if it were programmed directly into the addon. However, we don’t simply want to directly execute what gets entered into a custom string setting for multiple reasons:

  1. We need to be able to plug in the correct value for the sources depending on the item which the custom string is being evaluated for.
  2. The set of functions which are valid in custom strings are specific to TSM, and not necessarily part of the Lua language itself.
  3. We need a way to ensure that what the user enters is a valid TSM custom string, and not just some arbitrary snippet of Lua code.

This was previously accomplished by doing a very rough tokenization of the string followed by applying some rules and string replacements to the tokens to ensure the input was valid and create a valid, loadable snippet of Lua code to present the custom string. Tokenization is the process of turning an arbitrary string (i.e. the entire custom string the user enters into a setting input) into a series of known words (a.k.a. tokens). For example, the custom string “max(crafting, vendorsell)” gets turned into the tokens:

  • max
  • (
  • crafting
  • ,
  • vendorsell
  • )

In this tokenized form, we can then easily perform a series of validation checks, such as requiring that there is an equal number of left and right parentheses, and replace sources and functions as appropriate to turn this into a valid snippet of Lua code we can execute:

function(helpers, _item)
	local result = helpers._max(
		helpers._priceHelper(_item, “crafting”),
		helpers._priceHelper(_item, “vendorsell”)
	)
	if not result or helpers.IsInvalid(result) or result <= 0 then return end
	return result
end

Above is a simplified version of what the code generated by the original custom string implementation would look like. It looks very similar to the custom string we started with, but the sources have been replaced with function calls to look up the specified price for the item being evaluated, and the max() function call has been replaced with a call to a max() function TSM defines and provides.

This implementation has suited TSM well for a very long time, but has a few key limitations. First of all, the amount of validation that can be performed on a list of tokens is fairly limited and isn’t robust enough to catch all possible syntax errors. Having more robust processes in place for how custom strings are parsed will also enable better user-facing error messages and debugging functionality. Also, there’s no chance to perform any optimization on the generated code with this implementation. For example, if a custom string references the same source multiple times, the generated Lua code will look up the value of that source each and every time, with these lookups accounting for the majority of the computation time needed to evaluate custom strings in general.

New Implementation

The new implementation works much more like a traditional compiler and consists of 4 high-level steps: tokenization, abstract syntax tree generation, optimization, and code generation. We’ll use the following (somewhat contrived) custom string to demonstrate this process:
ifgt(crafting, 50g, crafting + 150% crafting, 10g + 5g)

For reference, here’s what that custom string would be converted to using the old system:

function(helpers, _item)
	local result = helpers._ifgt(
		helpers._priceHelper(_item, “crafting”),
		500000,
		helpers._priceHelper(_item, “crafting”) +
			1.5 * helpers._priceHelper(_item, “crafting”),
		100000 + 50000,
	)
	if not result or helpers.IsInvalid(result) or result <= 0 then return end
	return result
end

Tokenization

The tokenization step is similar to the prior implementation, but much more in-depth. It is responsible for taking the input string and converting it into a series of well-defined tokens which can then be more easily processed further. A token can be thought of similar to a word in an English sentence. It’s the smallest discrete unit which is meaningful by itself. The tokenizer works by methodically going through each character of the input string and separating out distinct tokens which are often separated by whitespace or formatting characters (i.e. parentheses or commas). In the case of our example (ignoring whitespace tokens), the tokenizer splits up our custom string into the following tokens:

ifgt ( crafting , 50g , crafting + 150 % crafting , 10g + 5g )

In addition to just splitting up the string into tokens, the tokenizer also classifies each token based on their type:

  • FUNCTION – ifgt
  • LEFT_PAREN – (
  • IDENTIFIER – crafting
  • COMMA – ,
  • MONEY – 50g
  • COMMA – ,
  • IDENTIFIER – crafting
  • MATH_OPERATOR – +
  • NUMBER – 150
  • MATH_OPERATOR – %
  • IDENTIFIER – crafting
  • COMMA – ,
  • MONEY – 10g
  • MATH_OPERATOR – +
  • MONEY – 5g
  • RIGHT_PAREN – )

The tokenization step takes care of a lot of the ambiguity of string parsing, and we now have a well-defined representation of the custom string which can more easily be worked with programmatically. The next step is to take this series of tokens and convert it into an abstract syntax tree, or AST for short.

AST Generation

An abstract syntax tree (AST) is a tree-based (per the name) form which better represents the structure of the custom string and the discrete operations (i.e. mathematical operations and / or function calls) which it consists of. A description of the algorithm for transforming the list of tokens into an AST is beyond the scope of this blog post, but we’ll visualize it with our example string, starting with a slightly more formatted version of the original string:

ifgt(
	crafting,
	50g,
	crafting + 150 % crafting,
	10g + 5g
)

In this form, we can visually see that this example has an outer “ifgt” function which has 4 parameters. Its 3rd parameter consists of 2 separate math operations (“+” and “%”). The 4th parameter of the outer “ifgt” also consists of a “+” math operation. Each of these distinct functions and operations is turned into a node in the tree, with its arguments becoming its children:

This AST form is helpful for doing a few things:

  1. It abstracts away all the original syntax of the custom string with the order of operations entirely determined by the location of a node within the tree. For example, we can look at the tree and see that we must calculate the “%” operation of “150” and “crafting” before we can calculate the “+” operation, as the result of the former is a dependency of the latter.
  2. We can easily validate that functions are being used properly. The “ifgt” function can only take either 3 or 4 parameters. We can easily see from this tree form that the “ifgt” node in the tree has 4 direct children, and therefore the “ifgt” function has a valid number of arguments. One thing worth noting here is that TSM stores enough information in both this tree form and the token list in order to reverse the process back to the original string form. This allows TSM to point the user at exactly where their custom string is invalid, which is one of the main benefits of this new custom string implementation as a whole.
  3. Lastly, we can perform very well-informed optimizations based on this AST form, which we’ll dive into next.

Optimizer

Now that we have the custom string represented as an AST, we can methodically go through it and look for opportunities where we can optimize it without changing the result. There are dozens of different optimizations which are attempted at this stage, but for this example there are just a few which apply.

The first optimization we can perform here is to realize that taking “150%” of something is the same as multiplying it by 1.5. Multiplication is generally a lot easier to reason about than a “%” operator, so we’ll update that part of the tree accordingly:

If we zoom out one level from that new multiplication, we can see that we now have the expression “crafting + (1.5 * crafting)” which, using some simple rules of algebra, we can rewrite as “2.5 * crafting”, or in AST form as:

Lastly, we can see that the 4th argument of our “ifgt” function is simply adding two constant values together, which can be precomputed to just “15g” with our optimized AST looking like this:

Although this was a relatively contrived example, we were able to reduce the number of nodes in the AST from 11 down to just 7 just through some simple optimizations. Most importantly, we were able to get rid of one out of three of the “crafting” source nodes. These optimizations mean that TSM users can format their custom strings in a way which makes managing them as easy as possible rather than worrying about trying to write them in an optimal way. The addon itself will take care of all the optimization.

Code Generation

The last step is to turn our optimized AST into Lua code that can be executed by the addon. Compared to the previous custom string implementation, having the AST allows us to do this in a much more controlled and optimized way. For example, we can cache intermediate values so we only need to compute them a single time, even if they are used multiple times within the custom string. The generated code for our example custom string (edited for readability) is as follows:

return function(itemString, helpers)
	-- Locals
	local res_ifgt_1901082 = nil

	-- Code
	local var_crafting = helpers.GetPrice(itemString, "crafting")
	if var_crafting == helpers.INVALID then
		return nil
	end

	if var_crafting > 500000 then
		res_ifgt_1901082 = (2.5 * var_crafting)
	else
		res_ifgt_1901082 = 150000
	end

	return helpers.ValidateResult(res_ifgt_1901082)
end

This code very closely matches our optimized AST, and has many optimizations when compared to our original custom string and the previous implementation. Most importantly, we only need to look up the “crafting” source for the item a single time, and then are able to reuse that multiple times within the generated code. One important optimization that’s not shown here is that we only evaluate sources as they are needed, and at most a single time. For example, if “crafting” was referenced in the “else” part of the “ifgt”, it would reference the same “var_crafting” variable rather than re-computing the crafting cost for the item.

Conclusion

Custom strings are one of the most powerful aspects of TSM, and we continue to dedicate significant time to improving their functionality and efficiency. With these under-the-hood improvements we made in 4.13, we are able to make the process of writing and debugging custom strings much better for all TSM users, while also improving their performance.

TradeSkillMaster 4.13 – Supporting Dragonflight 10.1.5

Following on from our last update with a recap of what was done and was to come in TSM in support of Dragonflight professions and systems – we’re pleased to share further improvements as well as the release of TSM 4.13!

Recent Dragonflight Profession Updates

Rounding out the last releases in TSM 4.12, let’s recap what has been introduced since the start of the year and our last blog update.

  • Updated milling / prospecting / disenchanting rates for Dragonflight reagents.
    • Given the significant variations of profession knowledge and equipment we gathered a lot of data to represent averages across the board as best as possible.
  • New quality selector to easily switch between different target qualities for a given crafted item, as well as easy access to add finishing reagents.
    • Selecting a different quality will automatically adjust the reagents selected to meet the selected quality, and is further enhanced in 4.13!
  • Quality reagents can now easily be changed by clicking on the material row in the material list
    • Seeing each quality of the reagent for at-a-glance pricing info and inventory quantity helps you decide what to use

New improvements in 4.13

With the launch of patch 10.15 we are introducing version 4.13 of TSM with a number of improvements to Dragonflight profession support and other additions we think you’ll love.

4.13 Profession Updates

Iterating on the things introduced in 4.12, we have continued to improve profession support including:

  • Adding support for Dragonflight salvaging for milling and prospecting
  • Adding support for crafting inspiration to show potential cost and profit of a crafted item that is not guaranteed
  • Improving the Optional Reagent flow further from the modal introduced in 4.12 to an in-line editor when a recipe is selected, allowing quicker access to add embellishments and finishing reagents

Region-wide support in 10.1.5

With the introduction of direct trading between any characters for gold and items on any realm in a given region, we are also very excited to expand on TSM cross-faction support added in 4.12 to cover region-wide tracking and reporting.

In 4.13, and if the option is enabled in the TSM General Settings, addon information such as inventory and accounting data will be available on any realm in your region. That way you can reference what you paid for an item and see how many you have on any realm from any other realm.

This will also include gold quantity and gold history, so your TSM Dashboard will cover all realms in your region if you’re actively using the addon on multiple realms!

To manage Accounting Data, the option to remove old sales and purchases has been modified to support an input instead of a predefined list in the Accounting Settings. If you’re making use of this feature and experience some performance issues it’s recommended to purge older data (which won’t affect your gold charts).

Custom String Performance Enhancements

Some TSM users have created some monster Custom Strings for all kinds of Operations and material prices. We’ve spent some time optimizing how Custom Strings are calculated to improve performance for very large or complex Custom Strings in very large profiles. This should also reduce the number of errors encountered when using TSM with some content creator profiles.

Grouping pets by level

We are also introducing ‘item level’ grouping for Pets in 4.13, where the pet item level translates to the level of the pet. When you’re grouping items by ‘item level’ they will be treated differently when shopping and posting through operations vs pets of other levels that are not grouped for example.

Dragonflight Update

In this post, we’re going to dive into where we’re at, what’s coming soon, and where we’re going with regards to supporting gold making in Dragonflight. I’ll start things off here by apologizing for not being as active as I would have liked with communicating this information sooner. TSM was definitely not as ready as I would have hoped for the changes in Dragonflight. There are a variety of reasons for this, which mainly boil down to not having enough development hours to spend and underestimating just how much of TSM would need to change to support the new profession changes. I understand people’s frustration with the state of TSM in Dragonflight so far, and am committed to improving things going forward. However, this will take some time.

Current State of Things

With that said, let’s talk about what changes we’ve made so far, and where things are at as of the time of writing (v4.12.20). The following things are working today:

  • For crafted commodity items, the crafting cost and profit will accurately display in the tooltip, and be available for use in custom strings (i.e. in Auctioning operation prices).
  • Quality, optional, and finishing materials can all be added manually to a recipe within the Crafting UI and then either crafted directly or added to the queue. Gathering then supports collecting all the materials as normal.
  • Restocking your groups based on Crafting operations is fully supported for crafted commodity items.

Coming Next

Here’s a list of things that are actively being worked on and can be expected to be released in the very near future:

  • Crafting cost and restocking support for quality crafts which produce gear of varying item levels.
  • Updated Destroying data for DF Milling, Prospecting, and Disenchanting.
  • Crafting UI support for selecting materials for and crafting recipes that don’t have fixed materials. This includes things like Mass Milling, Mass Prospecting, Recrafting, etc.
  • Improved UI for manually selected the target item for a quality craft, and having TSM automatically populate the cheapest set of quality materials needed.
  • Improved UI for selecting optional materials (including quality mats and finishing mats).
  • Display of crafting stats for quality crafts such as expected quality, inspiration, multicraft.
  • Support for taking inspiration into account when calculating the crafting cost.

Longer-Term Roadmap

There are a few other areas of TSM which definitely need some love and will be revisited in the not-too-distant future, which I thought I’d also call out here:

  • Improvements to how TSM handles buying commodities from the AH.
  • Item cache rebuilding performance improvements.
  • Account syncing performance improvements.

Crafting Cost

I thought it would be helpful to give some details on how we’re approaching crafting costs in Dragonflight. As we all know, Dragonflight greatly expanded the complexity of crafting things with professions. At its core, each recipe has a difficulty which gets matched up against your skill level to determine the result. That skill can then be modified with higher quality materials and various optional materials, as well as by proc’ing inspiration. In order to calculate how much it costs to craft something, TSM currently considers all the potential qualities of materials which result in the desired item being crafted.

As a hypothetical example, let’s assume there’s a potion which takes 1 herb and 1 vial to craft, with each of the materials being 1 of 3 qualities, and the recipe producing up to quality 3 of the potion. Let’s assume that using quality 3 of both materials results in quality 2 of the potion being crafted, and any other combination of materials results in quality 1 of the potion being crafted. To determine the crafting cost of the quality 1 potion, TSM will determine that there are 4 possible ways you could craft this item, from the 4 possible combinations of quality 1/2 of the herb, and quality 1/2 of the vial. TSM will then pick the cheapest of these options and use that as your crafting cost. Going further, it’s impossible to craft quality 3 of the potion, so TSM will treat that as an item you don’t yet know how to craft, and not display a crafting cost. Similarly, if you weren’t able to get your skill low enough to craft quality 1 of the potion anymore, TSM would treat that as something you can no longer craft and not show a crafting cost.

Currently, inspiration is not taken into account. From the above example, it’s possible that you could craft quality 3 of the potion if you were inspired. It seems like this will be important for some recipes where it won’t be feasible to get your skill high enough to craft at the highest quality without an inspiration proc for some crafts. This is something that’s not yet taken into account, but is something that we will plan to support in a future update as mentioned above.

Also, the game allows you to mix and max different qualities of the same material to hyper-optimize your crafting cost. For example, a recipe may require 5 of an herb, but you may only need to supply 1 at quality 3 with the remaining 4 being quality 1 in order to craft the desired quality of the crafted item. This isn’t something TSM takes into account, and is not something we have plans to support, due to the exponential increase in computing resource it would require to consider all possible permutations. Calculating crafting costs is already an order of magnitude slower than any other price source in TSM, so we felt that requiring all of a given material to be at the same quality struck the right balance between maximizing profits and keeping things performant and manageable for the user (i.e. when listing materials needed to craft something and/or gathering them).

Wrapping Up

In conclusion, there is a lot of work to do to get TSM where we want it to be for Dragonflight gold-making. This work is ongoing and we’re committed to aggressively work through the list of outstanding features and push out new functionality and bug fixes as they are ready. We’ll check in again with an update as things progress. If you have any feedback on the current state of things or the upcoming features, let us know in Discord.

TSM Addon 4.12 released!

Following the brief beta test, TSM Addon version 4.12 is now fully released for Dragonflight!

Check out our previous blog post for a full recap of the changes you can look forward to:

If you are running the TSM Desktop Application, you will be automatically updated.

NOTE: It may take up to 12 hours for you to receive this new version due to expected increased server load with the Dragonflight launch.

If you encounter any issues, please join us on Discord for support and help.

Announcing TSM 4.12 Beta Test!

As the launch of the Dragonflight expansion draws near, we’re very happy to announce the beta test of the 4.12 addon update to TradeSkillMaster!

If you’re a TSM Premium member you will shortly be receiving an email inviting you to participate, along with instructions on how to get started.

Additional invites through our Beta sign-up page will be distributed on a First-In-First-Out system, so be sure to sign-up if you’d like to get involved! Otherwise, you can bypass the queue and get instant access by signing up to TSM Premium.

Let’s take a look at some of the major features 4.12 brings!

New Crafting UI

To support the upcoming Dragonflight Profession changes, we’ve taken a look at the overall design of the Crafting UI for professions. This approach will form the basis of the UI design improvements as the new expansion features are introduced.

You’ll notice primarily that the Group Restocking flow has been moved out of a sub-tab and is presented at the top of the Craft Queue. Selecting Restock Groups will show the Group Tree and from here you can select any Groups that have a Crafting Operation assigned to be restocked.

At the bottom of the Crafting UI there has been some rearranging of some of the familiar elements including the reagents required for the recipe, the crafting cost, rank, and any recipe status such as ‘Missing Materials’ or ‘On Cooldown’. This will accommodate the new Dragonflight features including profession skills, and reagent quality selection.

In the Craft Queue itself, there’s a new indicator showing when you’re not logged into a character that can craft that particular item.

Pricing & Data Updates

Korean & Taiwan Region Support

We’ve been working hard to expand the reach of TradeSkillMaster, and with recent improvements by Blizzard on the Community Auction House API data feed we are very happy to be able to include support for more WoW Regions in 4.12. TSM users in Korea and Taiwan will now be able to use the TSM Desktop Application to get pricing updates exactly the same as users playing on North American and European realms.

Head over to the TSM website to set up your realm now!

DBRecent Price Source

Over the years we’ve had a lot of feedback and discussion around the use of ‘AuctionDB Minimum Buyout’ to represent the material cost of reagents, and we have generally not recommended this approach for various reasons. However, we do appreciate that there is a middle ground between what is happening on the Auction House ‘now’ and what has happened on average over the last 14 days with the ‘AuctionDB Market Value’ calculation – especially when there are significant changes to the economy such as an expansion launch, or a patch affecting a sub-set of items and markets.

With that in mind, we are introducing a new Price Source called ‘AuctionDB Recent’, or DBRecent. This Price Source uses the same approach for ‘AuctionDB Market Value’ however it is only applied to the auctions contained in the most recent pricing snapshot. It is not an average over time, nor is it a single data point for one auction at the lowest price contained in the last price snapshot.

We hope this can create even more flexibility when determining the value of something, and allow for more input to any decisions you make while executing your gold-making strategy.

Pricing Trend Indicators

In addition to the new Price Source, we are introducing trend indicators in item tooltips where pricing data is available.

Trend Indicators

In this example, you will see two trend indicators. The first is beside Market Value which represents the trend when compared to the Historical Price, this trend indicator is enabled by default. The second is beside Region Market Value Avg which represents the trend when compared to the Region Historical Price, this trend indicator is disabled by default.

Improvements to Region Sale Data

Both Region Sale Rate and Region Sold Per Day have been given another decimal point to give more visibility on items that sell infrequently. Previously, an item that had a Region Sale Rate lower than 0.01 for example, indicating that fewer than 1 in every 100 auctions posted sells on average, would not have a valid value for this source. Now, this value can go down to 0.001 which would indicate that 1 in every 1000 auctions posted sells on average.

Retail Scanning Improvements

Since the launch of the region-wide commodities, Blizzard has not resolved several issues with addons interacting with the Auction House. One of these issues is failing to return Seller information during a Post Scan with an Auctioning Operation. With the launch of TSM 4.12, we have decided to workaround this issue by ignoring the Seller information for posted auctions, which should improve scanning performance, particularly Post Scans.

As a result of this decision, we are retiring the ‘whitelist’ and ‘blacklist’ features in Retail which relied on the Seller information being available.

Cross-Faction Support in Retail

Earlier this year, Cross-Faction support was introduced in Shadowlands. This means you can now leverage professions, items, and gold on any realm regardless of whether the characters are Horde or Alliance.

With 4.12 we have introduced support for inventory/item tracking between factions in the Ledger and item tooltips, profession and recipe data is considered if the recipe is known on the opposite faction, gold is tracked across factions including in the gold graph of the TSM Dashboard, plus the ability to see your opposite faction characters in the Alts dropdown in the Mail UI.

Custom String Editing UI

We’ve supercharged the experience of editing Custom Strings in Operations, with IDE-style syntax highlighting.

Syntax highlight, invalid price source, and unbalanced parentheses

This new modal will give much more information when things go wrong like typos, unbalanced parentheses, missing values, or unexpected characters when building out a Custom String. It also includes automatic white-spacing for easier reading and clearer indications that the input is saved without accidentally closing/escaping the editor.

Automatic white-spacing once saved

Performance & General Enhancements

Under-the-hood Improvements

We’ve taken a look through a lot of the TSM addon codebase and refactored things to improve the rendering of TSM Addon windows and make them even more responsive.

Commodity buying reliability has also been improved while browsing in general and during a Gathering search.

Group Management

Item tracking is now more intelligent and can distinguish between things that are grouped as ‘Base Item’ and those that are grouped as ‘Item Level’. This means you can now manage inventory for things like transmog that could be any item level due to scaling but is generally grouped by Base Item.

When searching the Base Group, we have introduced the Select/Deselect All option that is available in most other Group Trees. Additionally when selecting an option in the ‘Show ungrouped items as’ drop down menu while managing groups, this selection will persist when navigating around the TSM UI or closing and reopening it.

The Addon API has been enhanced to include an option for returning items that are in a given TSM group, check the Addon API documentation to learn more.

Getting Help

Please join our Discord server as per the Get Started instructions, to share your feedback on the 4.12 Beta or report any bugs and issues you encounter.

New Public Web API

We are excited to announce that we are launching a brand new public web API! If you aren’t sure what this means, you can probably stop reading here and just keep an eye on for new spreadsheets and other out-of-game tools from our favorite content creators and other community members.

Background

We launched the original version of our public API back in early 2016 to allow our users and 3rd party developers to integrate our pricing data in their spreadsheets and other out-of-game tools, websites, and programs. However, we took this version of our public API down at the beginning of Shadowlands as we worked behind the scenes on an extensive (and still ongoing) process of rebuilding all of the backend infrastructure and services which generate TSM pricing data and power our website and desktop application. As part of this, we’ve been working hard on bringing online a new version of our API which leverages this new infrastructure, and we’re finally ready to make it public.

Features

The list of API endpoints we are releasing today covers all the same AuctionDB pricing data you would find in-game from realm-specific min buyout and market value, to region-wide sale prices and rates. This includes both base items and battle pets. This should enable a wide range out-of-game website and tools to leverage TSM’s data to build some great user experiences for gold-makers.

Get Started

Read through the documentation here to get started. We can’t wait to see what you build!

TSM Service Outage March 2022

Actions to Take

Let’s start with the actions you are recommended to take as a TSM user:

  • All users: Update the TSM Desktop Application to the latest version (r410)
    • Your app should update automatically, but you can download it here manually
  • All users: Re-select any Burning Crusade Realms previously configured on the Realm Selection Page for pricing data
  • All users: Enable any Group Notifications previously selected on the Deal Notifications Page

Background

Early in the morning US time on Friday, March 25th, we were notified by our hosting provider that the hard drive containing our primary database died, and they were unable to recover any data from it. After a ton of work, we have since been able to restore all of our services. However, there was some data which we, unfortunately, were not able to recover, which is listed as follows:

  • Premium user addon backups
  • Desktop App realm selections for Burning Crusade realms
  • User configuration data for Group Notifications
  • Some other user account-level and user configuration changes between 4am PDT on March 24th and when our website was brought back online including new accounts and any other changes made on our website during that time

What we Did

We take regular backups of our database and quickly worked to restore from those backups on Friday. Unfortunately, we realized that our backup process had not been updated to include database tables related to Burning Crusade Classic. Additionally, a configuration issue with a script that manages Premium user addon backups also resulted in all backups being lost as a result of this outage.

On Saturday we brought the website back online in a read-only mode as we continued to try to restore some additional database data from a few raw database files which had not been corrupted. After much work, we were successful at restoring some additional data which we otherwise didn’t have proper backups of, but unfortunately, this still left us at the list above of completely lost data.

Naturally, the Premium user addon backups are the most critical. Working with a manifest of the missing files, we modified TSM Desktop Application (version r410) to re-upload any backup files that match the name from the recovered manifest. While we realize this does not guarantee a full recovery of all previously synced backups we have been able to get coverage of a majority of recent backups stored locally to Premium users.

With this in mind, we have set up a dedicated contact address that is available to Premium users who may be missing an important backup that was not captured in the re-upload efforts. We encourage those to reach out to [email protected] and we will explore any available options to make this right.

Additionally, with new resources available along with the protections and processes described in the next section of this post, we have removed the limit on the number of Premium user addon backups that can be marked as ‘saved’ and stored by TSM in the cloud going forward. This means Premium users no longer have to decide which backups to keep available long-term and all backups will be saved indefinitely.

Finally, if you created an account or reset your password on March 24th or March 25th and are having trouble logging in, it is recommended to re-create your account or reset your password again.

Learnings / Improvements

We’ve only had one outage like this in the past, but this one is significantly more serious given the loss of user data. We do take seriously the impact of this event and are working hard to correct things in the short-term, as we discussed above, as well as make immediate changes to prevent this from happening again in the future.

First and foremost, we’ve enabled multiple additional levels of redundancy and protection around our Premium user addon backups. This will prevent them from being irreversibly lost in the future, along with providing a mechanism to quickly recover them if any issues arise. This will have a non-negligible impact on our infrastructure costs, as it is certainly a non-trivial amount of data, especially with the change we’re making to not limit the number of backups we store for people but is the right thing to do given the importance of this data.

Next, we have addressed the gap in our database backup process which resulted in some user configuration data being lost, so any similar event in the future will just be a matter of quickly restoring the backups we have. We’ve also built a number of additional tools to help us recover any lost data in the future, although, of course, we hope we’ll never need to use those.

Lastly, the affected database has been running since 2016. A lot has changed since then in the server infrastructure space, and we’ve been steadily working over the past year to reimplement and migrate our backend services and infrastructure to a more modern, scalable, and maintainable architecture. Most of this has been behind the scenes and is already supporting more user-facing things like Ledger and all of the AuctionDB data which is downloaded by the desktop app, but this outage has definitely raised the importance of moving more things over to this much better system as quickly as possible.

Closing Thoughts

To wrap things up, we’d like to sincerely apologize for the inconvenience caused by this downtime and for the loss of data that occurred. We are looking forward to doing better and further improving the software and services we provide moving forward, and appreciate your continued support and usage of TSM. If you would like to share any thoughts, questions, or feedback – please feel free to share them in the #discussion channel on our Discord server.

Anatomy of a TSM Release: Feature Selection

In this series, we are detailing what goes on behind the scenes to ship a new major TSM addon release. Specifically, we will be describing how version 4.12 of the addon is being built, from deciding what features will be included, to publicly releasing it. This first post will focus on how we decided what features will be in version 4.12 of the addon.

Selection Criteria

There is no end to the list of things we want to implement into TSM to make it the best gold making addon possible. We have curated a list of improvements we are considering on our public roadmap, so any discussion on what we prioritize for the next version starts by reviewing the list there. Our biggest constraint as a team (and, not to get philosophical, but in life as well) is time, but there are many factors we consider as we filter through this list. In no particular order, and with much less formality than they are described here, these include the following high-level points:

  • What portion of our user base would benefit from this feature? How much value will this feature bring to them? How much complexity would it add to the addon for new users or those who wouldn’t directly benefit from it?
  • Is this feature especially relevant to the current state of the game or the other projects being worked on within TSM (i.e. new pricing data functionality)?
  • What resources (design time, development time, dependencies on our web backend or desktop app, etc) are required to ship this feature?

Back when we were deciding what the focus of TSM 4.11 was going to be, we went through a ranking exercise to help us prioritize our list of high-level features we wanted to implement. This resulted in optional material support and custom themes being prioritized for 4.11, and also Gathering being called out as the next big feature to prioritize in 4.12. Therefore, when it came time to plan out 4.12, we went in with the idea of Gathering being the primary focus.

Digging Deeper

Once we’ve decided on the primary feature, in this case Gathering, the next step is to go one level deeper and figure out exactly what use-cases we want to better support. This is almost entirely influenced by feedback we’ve received around Gathering from our users over the years. The most challenging part of this is deciding what we are NOT going to do. Again, we are balancing how much of our time we want to spend on a given feature (and how long before we can release it) vs. how much value it will deliver to our users. For example, Gathering for multiple characters at the same time is not something we are going to be doing in 4.12 because it would require a ton of design and development effort to do it in a way which doesn’t add a ton of complexity that the user needs to manage. It would also provide relatively-low value when compared with the other features, such as supporting intermediate crafts from other professions. The result of this exercise is again reflected in our public roadmap under the “Prioritised Ideas” column. In the case of 4.12, it includes the following:

  • A bunch of general usability and UI improvements to the Task List and Gathering tab of the Crafting UI
  • Adding support for gathering from the bank in Classic
  • Support intermediate crafts on alts as a Gathering source
  • Improved connected realm support
  • Add a default “Farming” source which will track how many the player has acquired through a more manual means

This is a fairly ambitious set of features, but we are fairly confident in being able to tackle them within 4.12. Of course, things may change as we go along, and we may tweak this list as we go through the design and implementation process. However, this list is what we are currently working towards delivering. In addition to the major feature of Gathering, we also went through our overall backlog of feature requests and pulled out a few things which we thought would be easy to implement and/or especially timely to pull into 4.12. This includes things like persisting the “Show Ungrouped Items as” selection between sessions.

Next Steps

The next steps are split between design and development. On the design side, we are brainstorming what a better UI/UX for the task list and Gathering tab looks and feels like. On the development side, we are starting to implement some of the backend changes to support the new features, and knocking out other things which don’t have any dependencies on the new UI.

TradeSkillMaster 4.11 Released!

Version 4.11 of TradeSkillMaster is now available for general release! Your Desktop application will update your addon automatically, or you can download it directly from our site or your favourite addon manager.

More details on all the new features can be found in our beta announcement post, but here’s a quick reminder of the highlights for this release:

  • Full Optional Reagent Support, including Crafting Operations, Gathering, and applying them in the TSM UI
  • Improved item Grouping, with the option to group by ilvl while ignoring secondary stats:
  • Granular item Tracking, with inventory considered by ilvl/variation instead of only by name
  • Drag & Drop for items between groups:
  • Region Sale Data for Classic, BCC, and Battle Pets in Retail
  • Custom Themes with export/import options:
  • Various under-the-hood optimisations for scrolling tables with a lot of results and syncing data between accounts

Please do join us in Discord if you run into any issues or encounter a bug or error in-game, we’ll be happy to help!